My understanding of history is that the Empire was lost as a consequence of all those TPLACs and Asian dependencies realising that Britain wasn't powerful or rich enough to control them any more. Those damned slitty eyed nipponese blew away the idea of imperialist power and as a consequence India, Pakistan, Burma, Malaya, etc, began to flex their nationalistic muscles. South Africa, Australia, Canada, and New Zealand were already, effectively, beyond our control, and having been bankrupted by standing up to Adolf, and mortgaged to the Yanks, we couldn't afford to continue with the African adventure, nor retain the Caribbean and South American bits and bobs.
But hey, why not blame Labour for the invasion of Poland and Pearl Harbour whilst you're at it.