I'd say this is the result of huge investment bubble in IT companies, where nobody cared enough of efficiency of workforce.
Everybody seems to care of hardware efficiency, because it's very measurable.
For example, if CPU is overloaded, an extra virtual or dedicated machine costs a lot, or a daily script works 10 hours and may result in serious troubles -- everybody gets bothered.
But if you add complexity and devs must work a month instead of 1 week on a simple task -- this passes under the radar, because devs unlike CPUs are busy 100% time always. That projects start to take longer, probably gets attention, but many just considered hiring an extra guy.
This probably lead many to consider heavy and complex projects, like Airflow, worthy -- because the costs were not considered seriously, and apart them, the selling point (configuration with some clicks, and launching a task by a click), was in a favorable false dichotomy.
Everybody seems to care of hardware efficiency, because it's very measurable. For example, if CPU is overloaded, an extra virtual or dedicated machine costs a lot, or a daily script works 10 hours and may result in serious troubles -- everybody gets bothered.
But if you add complexity and devs must work a month instead of 1 week on a simple task -- this passes under the radar, because devs unlike CPUs are busy 100% time always. That projects start to take longer, probably gets attention, but many just considered hiring an extra guy.
This probably lead many to consider heavy and complex projects, like Airflow, worthy -- because the costs were not considered seriously, and apart them, the selling point (configuration with some clicks, and launching a task by a click), was in a favorable false dichotomy.