Yeah, that's a valid concern. Fortunately the read/write caching that S3DFS provides largely negates that problem. The caching is on the EC2 instance and so should provide great responsiveness to the app.
I was thinking more of the latency involved in making the roundtrip request from the app server to S3, which is probably on the order of tenths of a second. It would force you to rely heavily on a local memcached to achieve acceptable performance (which is probably a good thing anyway).
I'm reminded of a conversation with Facebook's CTO at Startup School, and he mentioned that they experimented with S3 for their photo hosting, but the latency of loading images from S3 made it unacceptable by their performance standards.
Facebook's 'young' and 'technical' performance requirements must be off the wall ;)
For the vast majority of apps, the latency will not be an issue. Especially since S3DFS has caching built in.
A good example is SmugMug who switches over to use S3 to serve out images when they are working on their own datacenter. Don McCaskill (their CEO and head programmer) says that switching back and forth was completely transparent to their users with no human-discernible latency. For more, see:
