The next major release of Tributary will support Avro, Protobuf and JSON along with the Schema Registry it will also bring the ability to write to Kafka with transactions.
But really you should get excited for DuckDB Labs to build out materialized views. Materialized views where you can ingest more streaming data to update aggregates. This way you could just keep pushing rows through aggregates from Kafka.
It is going to be a POWER HOUSE for streaming analytics.
Exactly. I have also been playing with DuckDB for streaming use cases, but it feels hacky to issue micro-batching queries on streaming data in short intervals.
DuckDB has everything that streaming engines such as Flink have; it just needs to support managing intermediate aggregate states and scheduling the materialized views itself.
Is this to be used in an analytics application backend sort of scenario?
I am familiar with materialized views / dynamic tables from enterprise-grade cloud lake type offerings, but I've never quite understood where duckdb, though impressive, fits into everyones use case. I've toyed with it for personal things, it's very cool having a local instance of something akin to snowflake when it comes to processing and aggregating on Big Data™ but generally I don't see it used in operational settings. For application development people are generally tied to sqlite and postgres.
It all does seem really cool though, I guess I'm just not feeling creative enough to conjure up a stream-to-duckdb use case. Feel free to bombard me with cool ideas.
Oh yes!! I've seen this a couple times. I am far from an expert in tributary so please take with a grain of salt.
Based on the tributary documentation, I understand that tributary embeds kafka consumers into duckdb. This makes duckdb the main process that you run to perform consumption. I think that this makes creating stream processing POCs very accessible. It looks like it is quite easy to start streaming data into duckdb. What I don't see is a full story around Devops, operations, testing, configuration as code etc.
SQLFlow is a service that embeds DuckDB as the storage and processing brains. Because of this, we're able to offer metrics, testing utilities, pipelines as code, and all the other DevOps utilities that are necessary to run a huge number of streaming instances 24x7. SQLFlow was created as a tool that I wish I had to for simple stream processing in production in high availability contexts :)
When I to the Naples Archaeological museum website, I get the following.
NOTICE
Il Mosaico di Alessandro proveniente dalla Casa del fauno di Pompei è in restauro.
Le indagini diagnostiche condotte negli ultimi anni hanno messo in luce una serie di criticità nello stato di conservazione del capolavoro.
Attualmente è in corso di ultimazione la prima fase di lavoro, che ha riguardato la progettazione e la realizzazione del sistema di movimentazione del mosaico, il ribaltamento di quest'ultimo e a seguire una ulteriore campagna di indagini diagnostiche e saggi che permetteranno di completare il quadro della conoscenza del manufatto. Seguirà la seconda fase di lavoro, che prevede prima il progetto di restauro e poi l'esecuzione dei relativi interventi sia sulla parte posteriore che su quella frontale.
Il pubblico può assistere alle attività del cantiere grazie a due distinti punti di osservazione appositamente predisposti nella Collezione Mosaici.
I wrote my master thesis on Acer C720 (2013) running xfce. I used some octave but I mostly needed it for latex.
These days I don't use it anymore because the screen is not that great but it aged incredibly well.
Nice post!
For completeness: Apache Arrow is also adding sparse tensors in c++ [1] and wrapping them with python [2] (although the documentation for Python might be a bit lacking at the moment).
reply