The way I'd do it would be to have a 5th official with access to the video.
If the referee hasn't seen something clearly or isn't sure, he can ask the video referee for advice and to review while the game continues, if required.
That way, the only impact on the game is that some decisions take a bit longer before the game is pulled back, but end up being correct more often.
The power remains in the hands of the referee, and the game isn't interrupted by requests for decisions to be reviewed.