A paper published by Databricks primarily focuses on why "Kafka sucks" and what if we make the source of truth a database and watch abstractions, then cases we miss with Kafka on resiliency and consistency can be solved.
It argues that modern pub-sub systems:
fail to achieve their goal of truly decoupling writes and readers
violate the end-to-end principle
expose an ad-hoc storage abstraction with a bespoke API and limited power
Marc put it extremely well. I agree with every single word of his thread. I should have applied a narrower and more specific definition of cache invalidation in the blog post. I apologize for any confusions it caused.
It argues that modern pub-sub systems:
fail to achieve their goal of truly decoupling writes and readers
violate the end-to-end principle
expose an ad-hoc storage abstraction with a bespoke API and limited power
relevant thread on X -https://x.com/BdKozlovski/status/1991892238335381524