LinkedIn recently announced that it transitioned from Kafka to Northguard.
Introducing Northguard and Xinfra: scalable log storage at LinkedIn [1] & LinkedIn: Stream Processing 4.16.25 [2]
Northguard doesn’t look like it’s been open sourced? I’d be curious to know how it compares to Apache Pulsar [0]. I feel like I see some similarities reading the LI blog post.
I haven't been following kafka for some years now, but i thought Linkedin were heavily invested in it. What happened? Also what happened to Confluent? Their team were ex-Linkedin members from what i remember?
NorthGuard looks like a clean sheet redesign of Kafka. The OSS Kafka community has taken a long time to implement things like KRaft, which addressed metadata scalability concerns by storing metadata in the brokers themselves (it used to be in a separate data store called ZooKeeper which was operationally complicated). NorthGuard also supports splitting and merging ranges of keys without repartitioning the entire existing dataset. The way records are assigned to partitions is a big problem in running Apache Kafka at scale because it requires predicting the key distribution and number of partitions ahead of time.
Confluent is 11 years old and IPOed several years ago. It was founded by 3 ex-LinkedIn people who originally designed Kafka. 2 of the 3 founders are still at the company. LinkedIn never used Confluent, Confluent was a company founded to sell an enterprise version of the open source project (and later a cloud version).
A new APT sources format "debian.sources" is announced with trixie. The now older "sources.list" format is still supported, but is likely to be deprecated in a future Debian release.
See below:
APT is moving to a different format for configuring where it downloads packages from. The files /etc/apt/sources.list and *.list files in /etc/apt/sources.list.d/ are replaced by files still in that directory but with names ending in .sources, using the new, more readable (deb822 style) format. For details see sources.list(5). Examples of APT configurations in these notes will be given in the new deb822 format.
If your system is using multiple sources files then you will need to ensure they stay consistent.
"apt modernize-sources" command can be used to simulate and replace ".list" files with the new ".sources" format.
Modernizing will replace .list files with the new .sources format, add Signed-By values where they can be determined automatically, and save the old files into .list.bak files.
This command supports the 'signed-by' and 'trusted' options. If you have specified other options inside [] brackets, please transfer them manually to the output files; see sources.list(5) for a mapping.
Oh nifty, I hand converted all mine a couple years back. It would have been nice to have that then (or know about it?). I do really like the new deb822 format, having the gpg key inline is nice. I do hope that once this is out there the folks with custom public apt repos will start giving out .sources files directly. Should be more straightforward than all the apt-key junk one used to have to do (especially when a key rotated).
Same. It took me a little bit to get used to it; my initial snap judgment was “this will be more annoying to create via scripting,” but then Ansible added deb822_repository [0] in 2.15 (shortly before Bookworm was released), and then it was no longer a concern.
Ooh, that's nice. Especially nice that it lets you specify both Suites: and Components: in the same stanza, so you don't have to repeat the rest of the line to add -updates and -backports suites.
[0]: https://www.newyorker.com/magazine/2018/12/10/the-friendship...