Yep. I'm not personally a huge fan of the kafka API — I think it's simultaneously too low level for normal users and too high level to deeply integrate into other systems (like stream processing engines), and requires a complex client library to use effectively.
We went with a simpler and more user friendly consume API, that also allows much higher levels of read parallelism (particularly important if you're using something like Workers, which parallelize well but aren't very powerful individually).
But we know many companies are invested in the Kafka ecosystem, and we want to provide an easy on (and if necessary, off) ramp for them.
Kafka libraries outside Java (and maybe Go) are such a disaster. I remember getting paged because all my NodeJS Kafka consumers were crashlooping from a segfault in Confluent's official Javascript client (wrapper around rdkafka C++). Why? Because Confluent changed some sort of telemetry setting on the broker. I also get segfaults in Rust using the most popular client out there (also an rdkafka wrapper) without any `unsafe` (although there were threads).
Definitely agree that every data system that doesn't need <100ms latency is moving to object storage.
> I do wonder if we will see an expansion of the s3 api to support more of these use cases
This is actually an area where I think we have a big leg up on folks building on top of S3. My team (which built K2) sits next to the R2 team, and we have the opportunity to co-evolve the products in mutually beneficial ways.
> every data system that doesn't need <100ms latency is moving to object storage
I think the opportunity extends below 100ms too, particularly given the existence of faster object storage tiers like S3 express or more recently GCS rapid bucket (both only offering single-zone durability, so still need to do quorum writes to get region-level durability as with standard tiers).
One of the tensions of course is how long to linger before flushing to object storage - you have to trade off directly between latency and cost of your API ops for PUTs.
When building the serverless offering of s2.dev (which is in a similar space, full disclosure!), we designed around stateful backend processes capable of constantly flushing multi-tenant objects (i.e., containing records from many streams), allowing streams to offer low ack latencies (~50ms p99 from same region) without blowing up the unit economics.
From my experience benchmarking S3 express, it is is faster, but not fast enough yet for many use cases.
An obvious disclaimer is that the word "enough" here is carrying quite the weight: I expect it to get better, and each has their own requirements. Do benchmark yourself and don't make expensive decision based on an HN comment.
Google PubSub is a great product. The primary benefit of K2 is cost, particularly for longer retention periods. Being backed by object storage means that we can store data extremely cheaply compared to disk backed solution, and we pass that on in our pricing.
Compared to self-hosted or cloud-hosted Kafka (e.g., Amazon MSK or Confluent), K2 is much cheaper, and fully serverless. There are no clusters to manage or scale, and consistent performance even as you vastly increase the amount of data.
The main downside is produce (and end to end) latency is higher (around 1s p99) than systems that rely on local disk replication, like Kafka.
So it's great if you're trying to move a huge amount of data around, or for use cases where cost is more important than latency.
Having built a hobo version of something similar (serving a minimal subset of the Kafka API on top of CosmoDB): there’s also the hybrid scenario where the cheap serverless streams are used for scalability and broadcast while a low-latency core is maintained on sharply reduced compute resources. An 80/20 approach that saves a lot and, in our case, reduced cluster (mis)management risks at the same time.
In our case BLOB and large document transfers were handled in parallel, merging them together through object storage is a highly appealing package. Great work!
The tail latency is higher than we would expect, but tbh our Consume API have work to do in terms of perf optimization. If it's not much to ask, can you share what's the max_records and the size of the stream that it was tested on? We are working on improving the read latency and you should be able to observe them in a week or so :)
Sure! There are definitely some overlapping use cases, and we've seen folks using/abusing queues for use cases that are more appropriate to something like k2.
Queues are great when you have a unit of work that needs to be completed, retried, and tracked individually. For example, a shop might need to call a payment processor API that can fail or timeout, and retry it until it succeeds, while polling on the frontend for the state of that particular message. With a queue, you can insert a message tracking that payment, and have a queue processor that keeps getting sent it until it succeeds or has failed too many times.
In a queue each item is its own thing that's important to someone, and queues give you APIs to interact with that particular item.
K2 is for moving large volumes of data around. Pricing is per GB, not per message. Records are produced and consumed in bulk, and what matters is that all records are processed, but no one is querying the state of a particular record. K2 also supports multiple consumers for the same record, and long term retention. For example, all of your applications may emit events when things happen, and those events need to be read by an alerting system, a system that durably stores them, and a system that uses them to build ML features.
One thing thing I am curious about. You mentioned polling for the state of that particular message (CF queue world). Is that really possible? I would have assumed it requires the developer to track the message using D1
My understanding: Distributed queues are generally good for when you have multiple workers processing chunks of work. Each queue item usually needs to be processed to completion exactly one time, so the queue provides the mechanism for the workers to coordinate state of each item at the item level (which allows for time-outs and re-tries if, say, a worker dies during processing, like if you were using spot instances for your worker pool)
Event streams are for multiple consumers, and can offer different guarantees. As far as I can tell, K2 is designed to ensure all consumers receive all events at least once (it's unclear to me whether this means they're continuously storing all events from the stream origin, or if older events age out at some point, or are dropped when they've been consumed by all known consumers).
Other types of guarantees with streams might be "at-least-once", "at-most-once", and "exactly-once" delivery, for different needs. Redis streams used to be at-least-once but it looks like they support all 3 use cases now. Some relational DBMSes also have the option to replicate by streaming their transaction logs to all servers in the cluster so each node maintains its own understanding of the database state (though stale reads can also occur in some/all? DBMSes that replicate this way, when a server is queried before receiving an update)
A stream can have up to 100 subscriptions. Each one has its own cursor, so every subscription sees every event independently. Events aren't dropped once they're consumed; they stay until the stream's retention period runs out, so you can replay or add a new subscription later.
Within a subscription, any number of workers can consume, with up to 128 batches in flight at once. Each batch is a lease owned by exactly 1 worker, and if that worker polls again it gets the same batch back. When it acks, the batch is done and the cursor moves forward. It can also nack to hand the batch back right away or extend the lease if it needs more time. If it does neither, the lease expires after 5 minutes and the same records go to the next worker that polls, as a new batch. That's where the "at least once" comes from, a late ack from the original worker is just ignored.
Yes! We originally built K2 to serve as the ingestion layer for Basin Pipelines [0], our stream processing product. We have a number of other teams building new products on top of it at the moment which I can't talk about yet :)
I've similarly spent a decade in the streaming space, started a stream processing startup, built three streaming platforms at various large companies... and I basically agree.
Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.
Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.
That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:
1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)
2. Avoiding expensive recomputations for periodic queries
I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.
(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)
During my time in the space, my pithy saying about the system I worked on [1] was that we could scale up or down. If you wanted to do streaming packet filtering with microsecond latency, we could do that. If you wanted to do complex analytics on structured data, we could do that. We did have deployments that "scaled down" and were more stream processing rather than streaming analytics. But analytics is by far the dominant use case, and SQL and relational databases are the better abstraction there. And for the stream processing cases, folks tend to stick to their existing lower-level stacks.
I'm increasingly of the opinion that "basically a data warehouse, but incrementally maintained materialized views under the hood as an implementation detail" (e.g. Materialize, Feldera, anything based on Differential Dataflow / DBSP) is a good point in the design space. I get that they're more complicated than batch, and for many orgs the low latency isn't necessary for pure analytics/dashboards.
I wonder if there's a place for such architecture in the operational path, not just analytics. If you squint, "this microservice reads data from this operational DB and passes it to this other microservice / puts it in a cache / sends it in response to a request" looks sort of like an "ad hoc, informally specified, bug-ridden, slow"[0] implementation of incremental view maintenance (in that a cache, or a service's data model in memory, is a sort of "materialized view" over the source data). I've seen some success in replacing a tangle of imperative-languages-and-network-requests with a declarative, incrementally maintained model in SQL, if the read latency can be kept under control.
I'm curious about the "some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache" part. could you expand on that?
[and yes, I do find these sort of problems interesting! :)]
I’m not sure you have to squint; an mview is just a cached query. The only difference between an mview and a cache is the “incremental” part of the equation.
That is, my only disagreement is the lack of gusto
the fact that traditional database infrastructure handles many people's needs doesn't really obviate dataflow style architectures. personally I think (3) composition is a real potential win. I would also note that streaming is a really great base on which to build a distributed database that looks more like Postgres on top.
I guess I'm mostly confused about the idea that streaming systems should be trying to supplant sql databases, or that a failure to do so implies that they don't have utility. it looks like the GP has done some really interesting work on automatic parallelism. that isn't pointless just because most people building operational systems still reflexively reach for PG.
That work is still relevant! It's just that end-users don't need to be aware of it. The position of the paper I submitted, which I basically agree with, is that "streaming" shouldn't need to be something end-users care about. It's something the system does based on needs.
Databases already have a dataflow style architecture: that's how they implement queries. Because SQL is relational, SQL queries become dataflow execution plans.
One way to think about the programming model I worked on is that it was like exposing a query plan API directly to users, instead of giving them SQL.
A project to watch in that space is Rama from Red Planet Labs [0]. It pushes further the trend of destructuring databases into a very flexible streaming engine and database.
It has no query engine, instead the platform offers powerful primitives to partition and distribute both data and compute, effectively destructuring transactions and query engine into microbatch and stream topologies.
I think the real value of such platform is in how it can replace tons of microservices and nosql databases into a single platform with a coherent model.
Replacing 1M LOC across 100 services and databases by one cluster with 10000 LOC is an operational dream come true.
Also I expect LLMs limitations to force us to resolve ourselves the operational complexity of IT, as I can't imagine letting LLMs multiply the number of services and databases like PRs on github. Platforms like Rama provide the necessary constraints to channel LLM output into a coherent whole easy to operate and observe.
It is indeed interesting, and it's from Nathan Marz, who was the creator of Storm. That was the first major open source streaming platform. It predated what I consider the default open source streaming platform, Flink.
Glancing through the docs, my main three reactions are: 1) It's sophisticated system which, as you say, effectively opens up the components of a database engine to be used as needed. 2) Folks will still want to eventually land their data in a "normal" data-at-rest storage format like Hive tables or Parquet. 3) Folks will still want SQL.
1. completeness is still a very desirable behavior for many use cases, but it's gonna be tricky to build these use cases on top of the new wave of IVM-like streaming systems, which provides no completeness marker.
2. periodic queries declared in non-streaming declarative form can and should be computed incrementally, but as far as I know, there's still not much mature support for automatic optimizations
ABC was originally designed for European folk music [0], not church music. The corpora as a result is largely fiddle tunes, particularly Irish (see for example https://thesession.org).
ABC started very simple, because most of the performance information for folk music isn't written down, it's inferred by the player according to the idiom of that particular tradition. As usage of ABC has grown it's gotten more powerful but still falls far short of formats designed for western classical, like MusicXML or Lilypond.
MIPS is increasingly common in rock helmets. Black Diamond (anecdotally the most common brand I see in the US) now has it in their higher end models, and Mammut as well.
Just checked my BD climbing helmet I bought few years ago (seems like Mips Capitan model based on design).
Considered these things a gimmick (any technical equipment bought has list of various tech used within, I generally ignore that by default since I have no idea what each means), happy to see move for more safety in this area. Even small steps matter.
I will climb in big heat wave we have here in Europe now this evening, more sweaty = more slippery on polished rock crags, risk is always not as far as we like to think.
Petzl makes better helmets than BD. They pioneered the ultralight foam kind with the sirocco that everyone (including BD and Mammut) copied. Yet they still don't care about MIPS.
(Disclaimer, I work at Cloudflare, but not on models)