Good article, very thought provoking. Here are some other dimensions that affect usefulness / value of adding LLMs to a workflow. LLMs can be amazing, or worse than useless depending where you land on these - regardless of seniority.
- are you working in a domain and language adequately represented in the training data? For example, it’s a lot easier to prompt an LLM to do what you want in a React CRUD website than a Swift app with obscure APIs from after the knowledge cutoff. No doubt LLMs will eventually generalize so this stops being troublesome, but today with o3-mini, R1 et al you need a LOT of docs in the context to get good results outside the mainstream.
- greenfield codebase or big hairy codebase full of non-industry-standard architecture & conventions?
- how closely does your code formatting align to industry standard? (needing bespoke formatting, line breaks at column N, extra lint rules etc is distraction for LLMs)
Codebases should be moved toward industry standards so LLMs can be leveraged more effectively. Could be a good (if indirect) way for mid or senior engs to help their junior counterparts.
- are you working in a domain and language adequately represented in the training data? For example, it’s a lot easier to prompt an LLM to do what you want in a React CRUD website than a Swift app with obscure APIs from after the knowledge cutoff. No doubt LLMs will eventually generalize so this stops being troublesome, but today with o3-mini, R1 et al you need a LOT of docs in the context to get good results outside the mainstream.
- greenfield codebase or big hairy codebase full of non-industry-standard architecture & conventions?
- how closely does your code formatting align to industry standard? (needing bespoke formatting, line breaks at column N, extra lint rules etc is distraction for LLMs)
Codebases should be moved toward industry standards so LLMs can be leveraged more effectively. Could be a good (if indirect) way for mid or senior engs to help their junior counterparts.