You're not being racist, but when "chinaman" is included in a negative description of someone, the two can get intertwined. I doubt anyone would be offended if I said my neighbour is a chinaman who cooks a fantastic chow mein.
I suggest "he's potless ****ing arsehole" as being less likely to offend.
HTH