Wrong - the Prime Minister is appointed by the Queen, on the advice of the Privy Council and at Her prerogative, as being the person most likely to command a majority in the House of Commons, usually the leader of the majority party. Therefore, by voting for a particular candidate in your constituency, you are endorsing that party's leader as a potential PM. If, however, the leader of a majority party in a GE loses his/her seat, then a nominee from that party's MPs would be selected.
The only 'recent' exception was Churchill in 1940, who was nominated by Lord Halifax and endorsed by Chamberlain despite not being the leader of the Conservatives.