I watch a lot more Rugby than Football but I cannot see loads of people coming down to Southampton from London to watch Rugby regularly thats all.
It is different for you as there are no decent teams in Hampshire so you have to travel as does a friend who lives in Basingstokes and is a season ticket holder at the Stoop.