Different contexts for different situations. Like you, I think if a stranger calls you "mush", it would be a bit of an insult, whereas if somebody you know well does, it's a term of endearment. I think it comes from the fact it's slang for "mouth". If a stranger called you mouth, you would be entitled to get a bit upset.