> deploying and testing on real world is much easier and faster than letting loose an ML model on it.
No this is what you're not getting. It is "doing the things I would do to deploy and test in the real world, but faster and in the background while I do other things", it is not "letting loose an ML model on it". This is the new capability. If you have any process like "do {action}, wait until {something}, check {something}, determine if it matches expectation", it is now possible to run that loop way more times in way more variants without either spending the time on it synchronously oneself or writing a script to do it. (If you do that specific action loop often enough, it's probably worth writing the script anyway, but that's also much quicker to do now.)
The AI doesn't need training on the domain, it just needs to be told "these are the things I would do, please do them for me and report back".
I'm sympathetic to not everything being nail-like, but I really think you're leaving a lot of chips on the table if you can't imagine any of this kind of action-check-evaluate loop you have that you could offload.
I'm saying you could do more things. Instead of waiting until a new thing is merged so that it can be deployed and waiting for monitoring to catch issues, you could be deploying pre-merge to testbeds, using different variants of the code or different configurations of the whole system or different inputs to exercise edge and corner cases. Maybe it's too complex of a system to set up that kind of testbed or simulation? Well, it's easier to do that now too! I can spin up and down environments, either with containers on my workstation, or in cloud deployments, that I would have not attempted in the past, because it would have taken me too long to set them up. But now it doesn't take that long, and I find it super valuable to be able to try more things out. (Cost is still a real constraint, I'm not saying that constraints magically no longer exist.)
Obviously I have no idea what your work looks like! But what I'm saying is that time savings are not just time savings. There can be a point at which the time savings bring you under an "activation energy" such that it unlocks a new capability, not just a speedup. And some of those unlocked capabilities can be directed toward improving the quality of software. And I think that's awesome and useful, is my prevailing point here. I won't claim that it will usher in an industry wide improvement in quality or anything, but for me personally, I'm making better software more quickly now, and I'm very pleased that I can do that.
No this is what you're not getting. It is "doing the things I would do to deploy and test in the real world, but faster and in the background while I do other things", it is not "letting loose an ML model on it". This is the new capability. If you have any process like "do {action}, wait until {something}, check {something}, determine if it matches expectation", it is now possible to run that loop way more times in way more variants without either spending the time on it synchronously oneself or writing a script to do it. (If you do that specific action loop often enough, it's probably worth writing the script anyway, but that's also much quicker to do now.)
The AI doesn't need training on the domain, it just needs to be told "these are the things I would do, please do them for me and report back".
I'm sympathetic to not everything being nail-like, but I really think you're leaving a lot of chips on the table if you can't imagine any of this kind of action-check-evaluate loop you have that you could offload.