Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Where I was a bit more than a year ago they hired a real expensive consultant who did a data lake project which wasn't finished by the time my team were told to use it, so we rolled our own according to that team's instructions and best practices (best practices were a huge deal at this company, everything we did was best practices). We exported data s3 under a particular structure and I built an ETL system around that and spotify's Luigi. Noone else on my team knew what ETL was, which made me feel old. We spent two, maybe three months on this. The BI team got their data and so did the marketing automation team.

But yeah, it's funny how these projects get complicated in larger organizations. Personally I would have rolled something even simpler on gnu/posix tools and scripts, in rather less than a month.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: