I don’t know if we could sufficiently prepare ourselves for such a world. It would seem almost as if we have to build it first so it could determine the best way to prepare us.
For one thing, we could try to come up with safety measures that prevent the most basic paperclip maximizer disaster from happening.
At this point I almost wish it was still the military that makes these advances in AI, not private companies. Anyone working on a military project has to have some sense that they're working on something dangerous.
