The primary reason I won't outsource this service is trust. I don't trust anyone with that data.
Access to my email allows for resetting all my passwords, seeing all my contacts, and tracking literally everything about my day-to-day business and personal life.
I hope all your email is encrypted, because otherwise every hop the email traveling to/from your server takes has access to all the data you hold so dear.
Actually most SMTP now talk SSL to each others, so while the hops (generally, just 2: sender and receiver) could also be compromised, the attack surface is smaller than one would think.
Not to dispute your claim, but how do you know this? Is there some sort of global, internet-wide survey of SMTP servers somewhere that supports what you say?
It's easy to tell which mail servers are using SSL by looking at your mail server logs. He probably only cares that most of the mail servers talking to him use SSL (which is also true for every mail server I have administered).
This is a much smaller attack surface than a centralized data store holding static e-mail archives and accounts for a large number of users, open to at-will offline browsing.
DNSSEC and SSL further reduce this attack surface.
Access to my email allows for resetting all my passwords, seeing all my contacts, and tracking literally everything about my day-to-day business and personal life.