If you have a directory that you want to expose, you can set that up. It doesn't have to be just ~/Downloads/Firefox. If you want to expose something like ~/Documents but deny access to ~/Documents/Private you can do that. With a little effort, you can probably even configure a helper utility that toggles access on and off dynamically with a status charm in the notifications area.
That sounds incredibly cumbersome, akin to things like umatrix or noscript (which I use but 99% of users would never touch or be able to correctly control).
