Parquet+iceberg stored on s3 as a base. That is solid enough. After that comes v... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

fifilura on Aug 11, 2024 | parent | context | favorite | on: Building data infrastructure that will last

Parquet+iceberg stored on s3 as a base. That is solid enough.

After that comes various kinds of caches, maybe postgres for frontend? Or something streaming?

But once everything is stored as files you gain the freedom to experiment or refactor.

yfontana on Aug 11, 2024 [–]

In many cases, that means gaining the freedom of creating a huge mess (speaking from experience)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact