I had the exact same thing happen to me in east street, except the guy said it was because he had to get back home to stoke on trent. Now I lived in stoke for three years, so I asked him what part of stoke, he said "hhrrrrmmm, the east side I think", so I said which one of the potteries is it then? He just glowered and stalked off.
And that said, I still felt slightly guilty, wtf?