Agreed, I've said this for yonks... we're 'too nice' a lot of the time. The usual counter-argument from people that disagree is that "it doesn't affect the ref's decision making", but I'm yet to be convinced that it doesn't have a subconscious effect, especially in 50/50 scenarios as you say.