Definitely take him as soon as possible, do the whole package, get him something from the shop, burger van, program etc etc..
I went to my first game aged 7 and I'm still here 33 years later.
Took my kids to the first games at St Marys when they were about 4/5,, one now supports man Utd the other supports Arsenal.