I wouldn't put it as simply as that. Basic human rights are, and always have been, fundamental to our way of life. Picking and choosing, on a whim, who is entitled to those rights goes against everything humanity stands for.
I doubt you'd be too impressed if a member of your family was incarcerated for 2 months for giving a terror suspect the time and being seen on CCTV doing so.