I don't know any Jews, never have and so I cannot weigh any decision based on experience. Nor have I ever encountered anybody being anti-Semitic. Anti-Israel yes, but not anything about the Jewish race.
Based solely on that and my own experiences of football fans then the term Yids, in my experience, has been no more than a term for Spurs fans.
I really doubt than many football fans who use the term know what Yids means are anti-Semitic themselves. Doubt many, outside parts of London, have even met a Jew!
All, of course, IMHO.