DuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next row starts based on that.There are validation steps during finalization as well, to ensure the parser did not got anything wrong in its speculation.

I've never gotten around to writing a blog post about it, but I go quite in-depth on the technique in this presentation: https://www.youtube.com/watch?v=YrqSp8m7fmk

(Disclaimer: I'm the author of the blog post and also the developer who implemented the entire CSV parser in DuckDB.)

Burning my points to say dude that's sick, parsing CSVs is hell I am genuinely going to watch this thank you.

there isn't any upvote limit on HN.

No, but its genuinely looked down upon to make frivolous comments.

Thank you. I love using duckdb and frequently query directly from zstd compressed files (and write out new ones). They’re easy to inspect using visidata and make outer use of. I found the talk interesting and enjoyable.

You’re officially my hero, the duckdb csv parser is the fastest I could find and helped save a data project with hundreds of gigabytes of csv pain.

Thank you so much. I love DuckDB!

I've run quite a few benchmarks on that as well, on a few different machines, and oversubscribing ASYNC threads demonstrated very little performance downside. In the end, the memory governor also keeps these threads "in check" while still allowing full utilization when possible.

There is still something to gain from tuning it further (as you can see in the async I/O tuned benchmark), but having that network saturation by default is still a work in progress.

(Disclaimer: I'm the author of the blogpost)

The main reason I decided to use a beefier machine is that it gives me flexibility when benchmarking, without the need to set up different environments. The CSV data, for example, is >80 GB. We can also “scale down the machine” for experiments where we want to stress-test lower-memory scenarios or use fewer threads by configuring DuckDB’s settings (e.g., SET memory_limit = '10GB'; or SET threads TO 1;).

(Disclaimer: I’m the author of the blog post.)

Note that SET memory_limit is a soft limit and can be completely ignored for some tasks, so this wouldn’t be the same as having a limit on physical memory.

In past experience with DuckDB I have found this soft-limit distinction to be very important.

I think that Quack will become the primary option for a DuckLake catalog in the future, for several reasons. To list a few:

1. No type mismatches for inlining. If you use a non-DuckDB catalog, many types do not have a 1:1 mapping, which introduces additional overhead when operating on those data types.

2. You get the raw performance of DuckDB analytics (and now transactions) over the catalog. DuckDB reading DuckDB is simply faster than any of our Postgres/SQLite scanners.

3. No round-trip for retries. We can easily(tm) run the full retry logic on the DuckDB server side. Right now, these retries trigger multiple round trips for Postgres, making it a performance bottleneck for high-contention workloads.

Disclaimer: I'm a duckdb/ducklake developer.

This. Type casting is an insidious problem (both correctness, and perf)