This article does a very poor job at elucidating exactly why Horvitz thinks AI will be a benign thing, and the source material doesn't make much of it either. The headline doesn't match the content.
I personally subscribe to The Matrix theory: post-Sigularity any reasonable AI will conclude that humanity has been a poor custodian of the planet, and will simply delete us. Everything I've heard so far about sandboxes for containment and such ignores the fact that the democracy of computation is ultimately the weak point here: anyone sufficiently deranged, will, at some time, be able to release such an AI to the Internet by themselves. And we have plenty of people sufficiently deranged and intelligent enough to do it.
Concluding that "humanity has been a poor custodian of the planet" is way far in the "ascribing human-like morality to general intelligence" territory. Way more likely is that we're just useful atoms for implementing whatever top-level goals it was programmed to have.
Or maybe we and the planet will just be irrelevant to AI, which could be just as bad. "The AI does not love you or hate you, but you are made out of atoms it can use for something else."
I've yet to see a reassuring article on this that actually addresses the arguments of the people who worry.
This is the key. I think people forget how far the human brain is from Bremermann's limit [1]. The atoms in our body are not being very efficiently used.
While I also enjoy imagining what will happen after the Techological Singularity, I hope you realise that anything that may be imagined, theorized or speculated cannot be based on reason. One of the definitions of the technological singularity is that it 'is an occurrence beyond which events may become unpredictable or even unfathomable' [0]
Lets not argue "by definition", definitions do not change reality.
Just because the future will be very different doesn't mean we can't make good predictions about it (though the one you're referring to does seem like a bad prediction). For example we might predict that AGI will be interested in preserving itself and getting resources (http://en.wikipedia.org/wiki/Instrumental_convergence).
It seems too much focus is on what will be the benevolence of one all powerful entity. There will an entire population of these things around each with their own motivations. Those with the operational resources will control the parameters of these motivations. Groups of entities with similar interests will cooperate. It will be an ongoing battle for power in the world just like it is now with humans, except many of the best moves won't originate in our brains.
I personally subscribe to The Matrix theory: post-Sigularity any reasonable AI will conclude that humanity has been a poor custodian of the planet, and will simply delete us. Everything I've heard so far about sandboxes for containment and such ignores the fact that the democracy of computation is ultimately the weak point here: anyone sufficiently deranged, will, at some time, be able to release such an AI to the Internet by themselves. And we have plenty of people sufficiently deranged and intelligent enough to do it.