Appy is technically right as we didn't play Stoke in the league until we were promoted to the Premier League in 2012. They went up 2007/08 season, the year we stayed up on the final day.
In terms of bogey grounds/teams, it is entirely superstitious and Saints have, in the past, suffered a lot from this, IMO.
Yeah times have changed in terms of our squad improving dramatically over the past few years but there were times where you'd go to Old Trafford, Villa Park, Anfield, Upton Park (maybe a bad example), Highfield Road etc and expect absolutely nothing.
Now the league is a lot closer and anyone can beat anybody but I still think bogey grounds still exist for teams, regardless of the current squad. It's like it's instilled within the DNA.