I have a similar experience. Sadly, not everyone who's shat on in the initial refusal has the wherewithal or support to challenge. I don't mix in the same kind of circles as Alex, so I don't know of any shirkers, but I know several people who have had to go through the ordeal of an assessment, then an appeal, before getting what they need.
I think some people just assume that benefits are being stolen by the greedy, and that everyone who needs them, gets them.