I'm not familiar with S3 on datalake setup. When replicating a db table to S3, w... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

wejick on July 14, 2024 | parent | context | favorite | on: Building and scaling Notion's data lake

I'm not familiar with S3 on datalake setup. When replicating a db table to S3, what format will be used?

And I'm wondering if it's possible to update the S3 files to reflect latest incoming changes on the db table?

mritchie712 on July 14, 2024 [–]

The file format is often Parquet. The "table format" depend on what data lake you're using (e.g. Iceberg, Delta, etc.).

If you know Python, here's[0] a practical example of how Iceberg works.

0 - https://www.definite.app/blog/iceberg-query-engine

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact