I agree with you that it's a complex and subtle area of law. For more color on this, it's helpful to keep in mind the difference between criminal liability and civil liability. Also, the concepts of "reckless endangerment", "negligence" and "due care." Factors which can change what may be "okay" to "not okay" include private vs public property, intent and, the ever present catch-all, reasonableness.

The likelihood of getting in legal trouble, as well as the potential severity of the trouble, can vary widely depending on where, how, who and other surrounding context.