I would recommend getting off M4 at travel south to Siemens Way. There is a Virgin Media call centre on the industrial estate and various car parks available for about £5. Park in one of these and there is a footbridge over the River Tawe walk along the river path - past the new training ground and you will see the stadium. You also arrive at the away end so no problems say 10 minute walk. If you want a pre match pint try Frankie and Bennies and there is a "beer tent" usually next to F and B's where services is quite quick. Although choice of drinks is limited.
Getting back you will get back on M4 relatively quickly using the reverse route. Hope this helps.