20-24 was 4,313,200 so a conservative estimate would be 6,000,000.
From my 'back of a fag packet' maths, if all age groups had similar % turnouts and followed the same voting patterns as the exit polls then remain should have won. So it stands to reason that most of the non voters were in the younger pro-remain age brackets (assuming I haven't ballsed up the sums or misinterpreted the data).