True. Just the action of pushing your head into someone is violent conduct.
Sometimes players try to stay on their feet in the box, stumble, but still get a shot away. You'd think the ref would see that the contact hindered the chance, thus having to decide if an advantage was gained. I understand though that this is much more of a grey area. It's too easy to go down.