You might very well think that but the picture that you think you see is not what actually happens.
In an earlier life I did a lot of research into television signal processing and transmission and was a representative on international technical committees and have several papers published at international conferences.
These referees are not ‘ some fat middle aged bloke’, they are as fit as the players and are regularly tested for their performance levels.