I believe Google's AutoML is attempting to answer these types of questions. It's obviously internal-only so others can't fork the research...but it has helped them invent new specific networks like "EfficientNet for EdgeTPU" [0].
I think humans can still invent new macro structures like CNN's...but humans are inherently shit at analyzing "what if we removed one neuron in the 2nd hidden layer?". The subtle tweaking is really best left to an automated recursion process.
Humans are better at seeing/inventing macro structures - such as adapting the unidirectional GPT to a bidirectional ELMO/BERT. After the invention, humans are generally pretty good at determining "whether" a network can be used to solve a particular task, although not infallible [1: Can BERT generate sentences from a prompt like GPT?]
But computers are once again often better at quickly determining whether which (ELMO, BERT, or GPT) perform better on a particular task for which they are all at least feasibly suited.
Sure, but auto ML has not in most cases panned out. The ML name for auto ml is neural architecture search, which is mostly useless these days. NAS has shown to not be any better than standard random search across a neural network architecture. I do not say this to disparage googles results, only that they came up with the networks they did by expending huge amounts of computational power.
I think humans can still invent new macro structures like CNN's...but humans are inherently shit at analyzing "what if we removed one neuron in the 2nd hidden layer?". The subtle tweaking is really best left to an automated recursion process.
Humans are better at seeing/inventing macro structures - such as adapting the unidirectional GPT to a bidirectional ELMO/BERT. After the invention, humans are generally pretty good at determining "whether" a network can be used to solve a particular task, although not infallible [1: Can BERT generate sentences from a prompt like GPT?]
But computers are once again often better at quickly determining whether which (ELMO, BERT, or GPT) perform better on a particular task for which they are all at least feasibly suited.
0: http://ai.googleblog.com/2019/08/efficientnet-edgetpu-creati...
1: https://ai.stackexchange.com/questions/9141/can-bert-be-used...