As said above, you can get it blocked - but also you have to top it up with a balance anyway so there's only a certain amount anyone can use with it. That probably sounds like a lot of hassle, but there's also an auto-top up feature you can use if it ever falls below £20.
Besides, it's on your wrist...it's going to be hard to fall off and if it did you'd notice it.