I think he could, could have said something like their values don't align, a Johnson cultist plunging the knife into Sunak at the last knockings isn't the embarrassment to Sunak that you might think it is.
I see it as more desperate for Starmer to want to associate with the nutter, it's a bit of a slap for the local labour party. A dignified 'no thanks' would have worked far better.