Because AI often replicates things much more closely than what fair use would constitute, and doesn't label sources like when you are quoting. And it's generally harmful for humanity, too.
I'm specifically interested in situations where ML products do not do simple reproductions of copyrighted material. I'm aware that it's difficult to even know the space of output and to "align" the model correctly.
Are we normally required to label sources when referencing other copyrighted materials, whether in songs or movies or otherwise?
Depends on how much of the source you used (which is in line with how fair use works, unless you're developing parody). Given that AI is using the entire source: yes.
As we know with scraping cases, the amount data and time also may play a role in determining fair use (think in terms of buffet ettiquite. "all you can eat" does not in fact mean "you can eat it all by yourself"). Funnily enough LinkedIn (owned by Microsoft) did argue successfully in court against scraping a website.
There are lots of things that are not simple reproductions that are not fair use.
If I take ten of your copywritten photographs and stack them on top of each other in Photoshop with transparency, the output is not a simple reproduction of your work. If I sold that for commercial purposes you would be upset with me and likely have a copyright case.
That's an obvious example, but my point is there aren't super clean-cut definitions for these things, and it's not settled case law yet which side current AI training and content generation falls under.
There is tons of human made art in contemporary art museums that is copywritten stuff re-arranged.
Most famously might be Andy Warhol's Campbell's Soup cans. You can find plenty more though. Product labels, magazine covers, pasted together as "art" even thought each part is copywritten.
