Legally you won't have any trouble until you do. Nothing is that cut and dried. However, things that might get you into trouble are military, government or police buildings and activity, and criminal activity. I have never had any real trouble and have very rarely had anyone interfere with my photography at all.