I once saw the explanation that when someone is looking through your window at your house you also close the blinds or even call the police even though you have nothing to hide.