Could is not the same as would. Immigration has been a matter of concern amongst the British Public for years and years and yet there have been no discernible attempts to restrict it from either the EU or rest of the world. I don't believe either of the main parties has any political will to do anything about it.
Yes, we have had a lot of immigration from the EU. But it does not follow that if we left the EU it would all stop. Anyway it would take a couple of years to untangle ourselves and rewrite all our laws in which time there might well be a mad rush to get in here before we pull up the drawbridge.