It still isn't a conflict.
I gave a scenario up there somewhere.
Say my firm sues a client for non payment of fees. The other person opposes. We have a small claim. The judge may have worked for my firm back in the day, may even have been a partner. There would be no conflict, and that judge could quite properly decide the case.
To the lay person, I get that it looks wrong, but a historic connection isn't a conflict.