Yes, when it's analysed it is 'stupid' but that's my point....your average man in the street sees things in a much simpler fashion - they see something that is quite close to White and call it white and they see other things that are various shades of colour and call them....erm...coloured.
If human beings were to analyse every aspect of how current day language has evolved then language wouldn't evolve organically in the first place.
There must be thousands of words in the English language that, if you analysed them, wouldn't technically fit their usage, but that is the beauty of language....it's not always scientifically 'correct'.