Isn't it 2 profiles of engineers? Those who make production code and those who work in prototypes?
One thing is to understand why your model isn't converging, another is how to scale it up...
A machine learning model is core to my company, and core to that is pulling in, cleaning, and working with large dataset from a variety of platforms. This means:
- excellence in being able to clean data at the column level for millions of data points
- knowing how to work with such large scale data in a time efficient manner. One of our newer hires worked at the Postgres/SSD level to optimize and got it to where he could produce a full set within 5 minutes. Before that it was once every few months.
Being able to do these things is a prerequisite to building even a prototype of a model, and it requires substantial real-world programming experience to deal with those complexities.
the data science people I've worked with in multiple VC backed companies were not engineers at all. They were either "data scientists" or had engineer tacked on to the name.
I'm continually exposed to new kinds of software engineering roles I never heard of at tech companies. (fwiw software engineer is sometimes still considered an inflated title.)
