I agree, it is absolutely ridiculous and if I thought that leaving the EU would stop it then I would be leading the OUT campaign despite the enormous damage that it would do to our trade.
This is one of many papers on migration to the UK, page 12 gives some numbers:
http://researchbriefings.files.parliament.uk/documents/SN06077/SN06077.pdf
More than 50% are non EU and nobody is doing anything to restrict that at the moment so I don't see why it would be any different outside the EU. As I said before, all the EU citizens who are already here are unlikely to leave, how would they be removed? In recent years some 200,000 people a year have been granted British Citizenship, they will probably not have been EU nationals because they don't need to. An EU national can only come here if they are working, studying or self-sufficient.