Yes, we are working on it! :) Taking some of the learnings from current experimental JSON Object datatype, we are now working on what will become the production-ready implementation. Details here: https://github.com/ClickHouse/ClickHouse/issues/54864
Variant datatype is already available as experimental in 24.1, Dynamic datatype is WIP (PR almost ready), and JSON datatype is next up. Check out the latest comment on that issue with how the Dynamic datatype will work: https://github.com/ClickHouse/ClickHouse/issues/54864#issuec...
five minutes! with some silliness groupby("name").agg(the math stuff). So their testbed (coiled, a cluster manager?) is meant to do exactly what the challenge requires. and it was data out of S3.
one thing that i caught there was that parquet serves the same purpose as CSV but for much bigger datasets and now i want to learn more about it.
Cloud network egress costs is one of the reasons why increasingly infrastructure vendors likely to incur high data transfer costs (database / data warehouse services, message buses, machine learning, etc..) are moving to introduce BYOC or "Bring Your Own Cloud" deployment model (Data Plane runs in the customer VPC - original Databricks deployment model), instead of only pure SaaS (Data Plane runs in the vendor VPC - e.g. Snowflake) in their cloud offerings.
Databricks now has fully hosted too, but Snowflake still did not go the other way to introduce BYOC. Newer entrants, however - RedPanda, AnyScale, ClickHouse... - are going the way of providing both BYOC and pure SaaS options in their cloud. There are many other reasons to prefer BYOC, legal, security, data privacy, but network egress costs is one of them.
ClickHouse is in particular well-optimized to take on real-time analytical workloads from BigQuery - both functionally and from a cost perspective.
For anyone wanting to try it yourself, there guide to follow when syncing or migrating workloads from BigQuery to ClickHouse, with some benchmarks and back-of-the-envelope cost comparisons:
https://clickhouse.com/blog/clickhouse-bigquery-migrating-da...
However if you really want to optimize data currently residing in Postgres for analytical workloads, as the original comment suggests - consider moving to a dedicated OLAP DB like ClickHouse.
What we ended up doing is maintain meta-data in Postgres but time series data is stored in ClickHouse. Thanks for making / working on ClickHouse. I appreciate it very much.
Indeed, ClickHouse results were run on an older instance type of the same family and size (c5.4xlarge for ClickHouse and c6a.4xlarge for Timescale), so if anything ClickHouse results are at a slight disadvantage.
Eh, c6a is also an AMD Rome which has worse memory bandwidth at the tails and weaker per thread performance than Cascadelake (c5). I don't understand anything about this particular benchmark, but I wouldn't compare them simply as "older vs newer".
Interesting project! Thank you for open sourcing and sharing. Agree that local and embedded analytics are an increasing trend, I see it too.
A couple of questions:
* I’m curious what the difficulties were in the implementation. I suspect it is quite a challenge to implement this support in the current SQLite architecture, and would curious to know which parts were tricky and any design trade-off you were faced with.
* Aside from ease-of-use (install extension, no need for a separate analytical database system), I wonder if there are additional benefits users can anticipate resulting from a single system architecture vs running an embedded OLAP store like DuckDB or clickhouse-local / chdb side-by-side with SQLite? Do you anticipate performance or resource efficiency gains, for instance?
* I am also curious, what the main difficulty with bringing in a separate analytical database is, assuming it natively integrates with SQLite. I may be biased, but I doubt anything can approach the performance of native column-oriented systems, so I'm curious what the tipping point might be for using this extension vs using an embedded OLAP store in practice.
* SQLite has rock solid support for transactionally persisting data to disk, which is a difficult and complex thing that stanchion gets almost for free. But the downside is fitting the architecture of stanchion into sqlite's way of doing things. So that is a long way of saying yes, the biggest difficulty was first learning how sqlite works and the implementing features to work with sqlite.
* It's a good question and certainly an area worth exploring. I don't know the answer, but doing, for instance, a comparison of power consumption and performance on a phone for the same use case between sqlite+stanchion and chdb running side-by-side with sqlite would be very interesting. To spitball some ideas of areas that may benefit stanchion: caching (both code and data, application caching and hardware caching), data sharing (no need for decoding and re-encoding either within the host application or between databases), and unified transactions (this one is a stretch). As you mention, chDB (and also DuckDB) benefit from having an architecture designed for analytics.
* As mentioned elsewhere in these comments, the sqlite virtual table system does have some limitations, so I think you're right when it comes to query performance. However, those limitations are limited to the way sqlite queries data in virtual tables, so I think stanchion can be competitive on data size on disk and potentially on insert performance.
I do plan to run and publish some benchmarks of stanchion against chDB and DuckDB in the near term. So far, I haven't focused on performance with stanchion, but that will be more of a focus going forward. Plus it's good to measure first and use that to track improvements. Stay tuned!
I'm a product manager, and I'm glad that many commenters in this thread agree that we are not all useless :) Of course, just like with any role, there are those that are good and bad at it, but I am glad most of you worked with good PMs and know the value such a person can bring.
I also saw astute comments from folks that recognize that if the product manager role does not exist as a distinct role, someone else in the organization has to take on the work of determining what to build and when, based on market and customers needs - usually engineering or product marketing. There is nothing wrong with that, btw, just different way to slice the same pie of responsibilities. Finally, several folks rightfully pointed out that a product manager (person accountable for building the right product for the market) is very different from a project or program manager (a person coordinating internal activities).
The only thing I'd add here is that product management roles can also be very different, depending on the stage of the company (early stage vs late stage), the type of product it builds (e.g. B2B vs B2C), and the industry vertical. Looking at the author's LI profile, he seems to work for larger companies, in the financial sector, in the UK. Perhaps in that world (which I perceive to be a fairly rigid, regulated industry), the role of the product manager is quite different from what I have personally experienced working as a PM in the Bay Area for early and mid stage startups.
I personally agree that real-time OLAP databases have potential to better serve workloads currently in Postgres or cloud data warehouses that need real-time ingest and analytical queries. And simplifying developer experience on top of that, so you don't have to learn about all the details of a powerful database, really speeds up developer velocity.
Good to meet you! I hadn't seen graphjson before, I'll check that out. I'm also a fan of tinybird, and I think we have similar goals of wanting to make it easier for people to adopt OLAP.
On the technical side, I've made different design decisions when it comes to ingesting and processing data. For example, after a table is created, you can post new JSON, with different columns, and we ingest it without needing a manual migration step. I also have a different treatment for JSON arrays, where we can split them into multiple Clickhouse rows (rather than using a clickhouse array.)
Philosophically, I think there is a lot of room for open-source software that also has an amazing UI and developer experience. I've been writing OSS for years and basically think this is the best way to build successful developer tools.
Surprised to hear it. I still get quite a bit of engagement on X, certainly have not seen a decline in the past year. Also, a few folks I follow on X, even if they claimed to "have moved to Fediverse", have all quietly come back and started to post on X again.
I joined Mastodon myself in the past year, and at least so far I have not been able to see much engagement there, though I am still trying to figure it out. The biggest issue I see is lack of cross-server search, aside from by tag, but also just not enough post activity. Maybe I am still not following the right people.
Variant datatype is already available as experimental in 24.1, Dynamic datatype is WIP (PR almost ready), and JSON datatype is next up. Check out the latest comment on that issue with how the Dynamic datatype will work: https://github.com/ClickHouse/ClickHouse/issues/54864#issuec...