One thing I appreciated about the Sandstorm architecture, and I wonder if you now consider this a mistake, is that each grain holds its own state, using ordinary files and possibly in memory (since IIUC, there's one instance of a grain running at any given time).
But I would have liked to use an architecture like this in a SaaS application. We may wish that applications were self-hosted, but SaaS is what people are used to, and there's a lot of money in it. An architecture for SaaS applications based on many small, stateful units, like Sandstorm grains, would be a refreshing alternative to the current norm of trying to make everything stateless and web-scale. And I think it's something you could have sold. I remember proposing something like this to you on an HN comment thread a few years ago, but maybe I didn't articulate it well.
The reason I wonder if you now see the statefulness of grains as a mistake is that Cloudflare Workers are going even further in the other direction. Whereas Sandstorm grains can be long-running, Cloudflare Workers are very short-lived. Of course, I realize that you aren't explicitly developing Cloudflare Workers as the successor to Sandstorm; it's a new product with its own requirements. Still, I wonder if your thinking on that part of Sandstorm has changed.
An architecture for SaaS applications based on many small, stateful units, like Sandstorm grains, would be a refreshing alternative to the current norm itof trying to make everything stateless and web-scale
Can you expand on this? The way I see things, the more of my stack that's stateless the better. That goes equally at the application level and at the distributed system level. There are so many advantages to designing things that way that it's hard to imagine going back to the stateful "bad old days".
So I'm curious to read a treatise about this alternative small, stateful units model you describe.
Statelessness is desirable in large-scale cloud architecture because it means you can auto-scale by just running more instances of the thing and load balancing, and you can fix any problem by just nuking the whole instance and making a new one. Also, storing data reliably in a large distributed system is an extraordinarily difficult problem which most people have no hope of getting right, so it's best to rely on an off-the-shelf database written by people who know what they're doing.
But these are problems that only exist because you're trying to create a mega-scale centralized application.
In Sandstorm's granular model, you don't have mega-scale applications. You have many distributed small instances. The programming model ends up being much more like desktop or mobile apps. In those environments, state has never been that big a deal. You don't need a database that can store and index petabytes of data; you can use something simple like sqlite or maybe even flat files. You don't need to think about machine failure, backups, etc.; that is the OS and/or device owner's problem. (Meanwhile, small-scale instances lead to clear data ownership which means that the owner can take responsibility for backups, and a distributed OS can potentially migrate grains around as needed to work around machine failures, transparently to the app.)
Another big problem with the stateless approach to big web apps is that off-the-shelf databases are often a terrible fit for specific use cases, but typically changing the database to suit your needs is not an option. Databases tend to be particularly bad at real-time updates, e.g. like in Google Docs where you can see other people typing. To implement Google Docs reasonably, you need a stateful server; you need for all users of the same document to land on the same server instance so that coordination and routing can occur in-memory. That's incredibly difficult to achieve in the stateless cloud architecture orthodoxy, but incredibly easy to achieve in Sandstorm's model.
> you need a stateful server; you need for all users of the same document to land on the same server instance so that coordination and routing can occur in-memory. That's incredibly difficult to achieve in the stateless cloud architecture orthodoxy
Exactly, that's what I was getting at. I'm sure Google has a lot of internal infrastructure and know-how for doing this. But outside of giants like Google and Microsoft, AFAIK, tools and techniques for developing stateful web applications that are reliable and scalable don't seem to be widely known. And the orthodoxy, as you aptly put it, is so strong that I'd venture to say that most of us don't even consider the stateful alternative. So for those of us who do think about it, it seems like the unsafe choice that we're better off avoiding. I hope that changes.
We might learn from the gaming dev community, where stateful servers are quite common. It's interesting to see that for example agar.io (web-based realtime game) started in C++ and moved to Erlang, both of which uncommon choices for web software, though the latter seems to be rising, particularly in the form of Elixir.
I'm wonder how much of those are actual problems and how much is FUD though.
At least, I remember benchmarks for nodejs a few years ago that showed a single instance handle a million concurrent, long-lived connections - on a single thread!
So if you're not aiming for Google-scale load, the field still seems interesting to explore.
For starters, I'll refer you to Sandstorm's own "how it works" page [1], which explains it quite well IMO. Basically, the Sandstorm modl works well for applications that consist of many small things -- documents, chat rooms, project boards, etc. -- each of which can easily run in a single process. Each grain can run in a self-contained process, with its live state being stored in memory, and using the filesystem for durability. When things happen within the grain, it can easily send out notifications to all connected clients, e.g. through WebSockets, because all clients are communicating with that one process (indirectly through a proxy, of course). Consider how much easier it is to implement real-time things, like chat and collaborative document editing, this way, compared to using message brokers, distributed pubsub systems, etc. As for scaling and high availability, as the page I linked explains, the platform takes care of that.
But I would have liked to use an architecture like this in a SaaS application. We may wish that applications were self-hosted, but SaaS is what people are used to, and there's a lot of money in it. An architecture for SaaS applications based on many small, stateful units, like Sandstorm grains, would be a refreshing alternative to the current norm of trying to make everything stateless and web-scale. And I think it's something you could have sold. I remember proposing something like this to you on an HN comment thread a few years ago, but maybe I didn't articulate it well.
The reason I wonder if you now see the statefulness of grains as a mistake is that Cloudflare Workers are going even further in the other direction. Whereas Sandstorm grains can be long-running, Cloudflare Workers are very short-lived. Of course, I realize that you aren't explicitly developing Cloudflare Workers as the successor to Sandstorm; it's a new product with its own requirements. Still, I wonder if your thinking on that part of Sandstorm has changed.