Have you honestly seen trouble at Saints games on the scale of Millwall, Leeds, Cardiff etc? The worst i've seen is V Pompey, and that was nowhere near the level you see at the previously mentioned.
Personally I think we're considered "high risk" just because of the sheer amount of people we take away from home nowadays. My brothers a steward at Yeovil and they were told to expect a pitch invasion from the Saints end. There was nothing planned at all, it was just because there were so many of us.