Did you read the article and follow the sources? If not, why not? Also, why spout nonsense about this being an 'alt-right blog'? Westerholm is a 'betongsosse', an old-style social democrat. Are they 'alt-right' just because they don't do identity politics?
Oh you craving down-voting cowards, afraid of anything which disproves your narrative. Speak up instead of pressing that stupid button, let us know what you think and why what I said is wrong if you can.
You certainly don't need to agree with me or anyone else, you have as much right to an opinion as anyone else but hiding behind that button just to try to hide a dissenting opinion just proves you're on thin ice. Use that button for things which are clearly beyond the pale, and/or violate the board rules, use your voice to show disagreement.
The entire post is evidence for foreign - Here Somalian - influence in the Swedish voting process with sources provided. Opposing identity politics is not 'identity politics'.
Reducing politics to an identity-based thing (like stereotyping people's politics based on their race or national origin) is literally what identity politics is. Opposing identity politics would be talking policy without bringing people's identities into it, or ascribing people's politics solely to their identity.
"They're not like us" is identity politics.
"They voted differently than us because of their identity" is identity politics.
Sorry friend, it seems you love engaging in identity politics, otherwise your political positions would not be gripes based entirely on people's identity. Your post is like claiming you don't discriminate based on skin color, then in the same breath, saying that people of a particular skin color are bad.
Soo… usearch? Its literally one header file (of what use to be strict c++11). Funnily enough that is what is used in the official duckdb-vss extension.
This is cool. My only worry is that the implementation complexity will prevent widespread adoption outside of maplibre. Although getting write support upstreamed into PostGIS might be all thats needed to make sure it trickles down into all the different tile servers. MVT is not the most efficient, but everything speaks protobuf and you can hack together a parser in an afternoon.
I've experimented a lot with vectorized encodings of geometries in DuckDB-spatial using the different nested types. You definitely do get very good compression out of the box if you already support a bunch of specialized lightweight compression algorithms. Simpler geometric properties are very fast to compute (e.g. area, length), but for anything more complex you usually need to do some pre-processing or conversion into an intermediate data structure (like creating a line-segment index for intersection checks, or a node graph for clipping) which dominates the processing time anyway. The cost of materializing the columnar format into a row-wise format and back again when doing joins or sorting is absolutely brutal on performance too, compared to just keeping geometries as serialized blobs that are easy to slice and memcpy.
That said, I do expect columnar encoding to work really well for rendering in the browser, where transfer speed is the big bottleneck. The paper mentions Arrow as an inspiration, but I wonder why the format isn't just based on (compressed) arrow in its entirety? Im not super up to speed on the arrow ecosystem but I know there's a couple of query engines that don't just use it internally on the CPU, but also to execute on the GPU. If you are going to decode and send over the data to WebGL, you might as well do the filtering/expression evaluation there too no? (and leverage the existing techniques/code/interop in the arrow world)
> My only worry is that the implementation complexity will prevent widespread adoption outside of maplibre.
I don't think the implementation is that complex. You may be underestimating the extent to which heavy users of mapping data already write their own informal bug-ridden versions of the transforms and representations standardized in this tile format. In fact, it is not uncommon for companies to actually be using multiple slightly incompatible implementations of these under the hood. The practical effect could be to actually reduce the amount of code being written to do this, never mind the compatibility bugs it would address.
A key caveat is that this format is explicitly optimized for visualization. It is not optimized for efficient geospatial or spatiotemporal analytics, which may not even have visualizable output in this sense. Formats optimized for analysis make a very different set of tradeoffs.
I think this is just cause it hasn't been implemented in spatial yet. DuckDB is currently going through a pretty big refactor of the way we glob/scan/union multiple files with all the recent focus on data lake formats, but my plan is to get to it in spatial after next release when that part of the code has stabilized a bit.
Im inclined to agree, but unfortunately a huge amount of the existing data and processes in this space does not assume a spheroidal earth and come provided with a coordinate reference system. Ultimately there are also some domains where you got data that you explicitly don't want to interpret using spheroidal semantics, e.g. when working with a city plan - in which case the map _is_ the data model, and you definitely want the angles of a triangle to sum up to 180.
Similarly I see it as it an inevitable that you will deal with a problem related to these issues, then have a massive blocker if you aren't already familiar with the details because debugging requires more than a cursory understanding for hard problems. In the alternative, you don't know the problems exist and the code is broken. I don't think it's as simple as abstracting away the entire concept, which is what I would say is too high level like I mentioned above. I don't know the right answer here honestly, I think it will be disruptive when someone figures it out
I replied to another comment, but I think a big part is that duckdbs spatial extension provides a SQL interface to a whole suite of standard foss gis packages by statically bundling everything (including inlining the default PROJ database of coordinate projection systems into the binary) and providing it for multiple platforms (including WASM). I.E there are no transitive dependencies except libc.
Yes, DuckDB does a whole lot more, vectorized larger-than-memory execution, columnar compressed storage and a ecosystem of other extensions that make it more than the sum of its parts. But while Ive been working hard on making the spatial extension more performant and more broadly useful (I designdd a new geometry engine this year, and spatial join optimization just got merged on the dev-branch), the fact that you can e.g. convert too and from a myriad of different geospatial formats by utilizing GDAL, transforming through SQL, or pulling down the latest overture dump without having the whole workflow break just cause you updated QGIS has probably been the main killer feature for a lot of the early adopters.
(Discmaimer, I work on duckdb-spatial @ duckdblabs)
I'm not the OP, but thank you for such a detailed answer. The integration and reduced barriers to entry you mention mirror my own experiences with tooling in another area, and your explanation made parallels clear.
Is there any strong reason to use GeoParquet instead of straight up parquet if all I'm interested in is storing and operating on lat/lons?
I'm curious if it compresses them better or something like that. I see lots of people online saying it compresses well (but mostly compared to .shp or similar) but normal parquet (.gz.parquet or .snappy.parquet) already does that really well. So it's not clear to me if I should spend time investigating it...
I mostly process normal parquet with spark and sometimes clickhouse right now.
Based on my reading of the GeoParquet spec, the main difference is that geometries are stored as WKB using Parquet's byte array type. Byte arrays can be delta-encoded. There is also some additional metadata stored like CRS and a bounding box.
When using EPSG:4326 lat/lons, I don't think GeoParquet would give you any benefits over just having separate columns (this is what I typically do, and it's plenty fast).
If you are using range requests to fetch only parts of parquet files at a time, you could potentially sort your data using a hilbert curve, which could limit the number of row groups that need to be fetched to execute a query.
> a big part is that duckdbs spatial extension provides a SQL interface to a whole suite of standard foss gis packages by statically bundling everything (including inlining the default PROJ database of coordinate projection systems into the binary) and providing it for multiple platforms (including WASM). I.E there are no transitive dependencies except libc.
and for the last twenty, not ten years, this is what PostGIS was pioneering, and also teaching everyone get used to. DuckDB was not something that people even knew in GIS world. I'm not even sure whether QGIS connects to DuckDB, perhaps it does for a while, but it sure knows Spatialite for very long and last, but not least - ESRI sure as f*ck still have not heard of DuckDB. This is already half the geospatial world out there.
This whole article is superb biased and its very sad.
no surprise I missed it, 3.4 was only released in 25.11.2024 г. which is like yesterday in ESRI terms, given price and speed of adoption of new versions (note: many .gov still use the 10.x/11.x branch).
which does not change my original statement that nobody cared about DuckDB for very long, while the whole server-side processing idea is largely based on PostGIS.
As someone unfamiliar with DuckDB but at least somewhat with geospatial tools (it's been a few years, though):
Dang - see, now that is seriously cool. The whole ETL shebang always was the biggest hassle, even with serious commercial tools, and the idea of a stable, all-in-one, ready-to-go layer is incredibly appealing.
It's just something the writer of the article should probably have at least mentioned when going full hyperbole with the title (hey, after reading this, it might actually be justified! :) ).
Rereading the article that focuses on the one thing that isn't a standout (the installation itself), though, I can't help but chuckle and think "worst sales pitch ever". ;)
I think a big part is that duckdbs spatial extension doesnt have any transitive dependencies (except libc). It statically packages the standard suite of foss gis tools (including a whole database of coordinate systems) for multiple platforms (including WASM) and provides a unified SQL interface to it all.
(Disclaimer, I work on duckdb-spatial @duckdblabs)
I tried out duckdb-spatial for a hobby project and my takeaway is the main thing it offers over geopandas is performance in batch processing.
If I can reduce my spatial analysis to SQL and optimize that SQL a little bit, duckdb will happily saturate my CPUs and memory to process things fast in a way that a Python script struggles to do.
Yes, the static linking is nice. It helps with the problem of reproducible environments in Python. But that's not a game changer IMO.
"Accessibility" is too often dismissed, yes you CAN do things with things, but getting people to do it is a craft and art. This is often where open-ish-core differs from the enterprise version of something too
Thanks! I've been wanting to add this since I first started out working on DuckDB almost two years ago but I finally managed to accumulate the time (and the skills required!) to finish it up over the summer. It still has a long way to go, support for indexes in extensions are pretty... raw, and we only push down constant filters into index scans (so no spatial index-join acceleration yet). But I think having a proper spatial index is one of those things that are kind of required to really elevate the spatial extension from being just a toy and I'm super stoked to work more on it during the next release cycle and all the new possibilities that it opens up.
I still need to update the docs and there's probably going to be a blog post on it in the future, but for now there are some more details in the PR https://github.com/duckdb/duckdb_spatial/pull/383
It does push down filters! Not sure if other nodes are pushed down, I think for now the database boundary is at the scan node, but you could add additional optimizer/rewrite rules in the postgres/sqlite/mysql extensions to push down other parts of the plans when applicable.
