As database builders and users, we’ve made talking about systems a lot harder on ourselves by conflating the ideas of replication, active-active, atomic commitment, and concurrency control.
- Replication is a technique used to achieve higher availability and durability than a single node can offer, by making multiple copies of the data. Techniques include Paxos, Raft, chain replication, quorum protocols, etc.
- Active-active means that transactions can run against multiple different replicas at the same time, while still achieving the desired level of isolation and consistency.
- Atomic commitment is a technique used in sharded/partitioned databases (which themselves exist to scale throughput or size beyond the capabilities of a single machine) to allow transactions to be atomically (“all or nothing”) committed across multiple shards (and allow one or more shards to vote “nah, let’s not commit this”). 2 phase commit (2PC) is the classic technique.
- Concurrency control is a set of techniques to implement isolation, which is needed in any database that allows concurrent sessions (single node or multi-node). Classic techniques include 2PL and OCC, but many exist.
When vendors or projects answer concurrency control questions with replication answers (which appears to be the case here), it’s worth diving deeper into those answers. There are cases where “Paxos” or “Raft” might be answers to atomic commitment or even concurrency control questions, but at best they are very partial answers and building blocks of a larger protocol. Databases that only support “single shot”/predeclared transactions can get away without a lot of concurrency control, for example, and might be able to do the required work as part of their state machine replication protocol. In general, I'd see using words like "Paxos" and "Raft" in the marketing for a database as a negative sign. It's not a fully reliable one, but it's often the least interesting part of the implementation and the choices the database is making.
To be extra clear, I’m not criticizing Aphyr here (the article clearly doesn’t conflate these concepts), but more pointing out what I think lies at the bottom of a lot of the issues we see with distributed database claims.
I think Aphyr helped a lot by creating this useful resource: https://jepsen.io/consistency which presents a clear classification of consistency models. I am not sure if talking about anything else in the context of distributed databases is reasonable.
That is one way (and a good one) of classifying consistency models and there relationship to isolation levels. But it's an incomplete one (e.g. there are linearizable variants of snapshot and repeatable read that exist that are not captured there). I'm a big fan of that stuff (and Aphyr's work in general), but that page is the beginning of a conversation and not the end of one.
You'll find many of those additional variants (e.g. strong session SI) in the linked papers, and they're cited extensively in the Jepsen reports as well. I just haven't had time to write up every single model--tried to stick to the major ones. :-)
> I am not sure if talking about anything else in the context of distributed databases is reasonable.
There's a whole world in distributed databases, and I suspect you'd agree that there's a lot of stuff worth talking about that isn't covered in your (excellent) work.
- Replication is a technique used to achieve higher availability and durability than a single node can offer, by making multiple copies of the data. Techniques include Paxos, Raft, chain replication, quorum protocols, etc.
- Active-active means that transactions can run against multiple different replicas at the same time, while still achieving the desired level of isolation and consistency.
- Atomic commitment is a technique used in sharded/partitioned databases (which themselves exist to scale throughput or size beyond the capabilities of a single machine) to allow transactions to be atomically (“all or nothing”) committed across multiple shards (and allow one or more shards to vote “nah, let’s not commit this”). 2 phase commit (2PC) is the classic technique.
- Concurrency control is a set of techniques to implement isolation, which is needed in any database that allows concurrent sessions (single node or multi-node). Classic techniques include 2PL and OCC, but many exist.
When vendors or projects answer concurrency control questions with replication answers (which appears to be the case here), it’s worth diving deeper into those answers. There are cases where “Paxos” or “Raft” might be answers to atomic commitment or even concurrency control questions, but at best they are very partial answers and building blocks of a larger protocol. Databases that only support “single shot”/predeclared transactions can get away without a lot of concurrency control, for example, and might be able to do the required work as part of their state machine replication protocol. In general, I'd see using words like "Paxos" and "Raft" in the marketing for a database as a negative sign. It's not a fully reliable one, but it's often the least interesting part of the implementation and the choices the database is making.
To be extra clear, I’m not criticizing Aphyr here (the article clearly doesn’t conflate these concepts), but more pointing out what I think lies at the bottom of a lot of the issues we see with distributed database claims.