(I gave a talk about this at CCC. The status quo is a reason why free software doesn't have some of the transparency benefits over proprietary software we might expect, if you fear some developer is attacking you or someone is attacking some developers' infrastructure.)
While that's true, I still know people who compile their own OS from scratch; I've gone through the experience of compiling my entire system from source via Linux From Scratch[0].
There are many thousands of people who have done the same thing, and this gives us a "web of trust". And if you don't want to trust anyone, you can always compile it yourself.
That's the difference: I have the freedom to learn and study anything and everything. It might not be likely, and it might not be convenient, but it is possible.
I'm not sure about reading, but several distributions (notably, Gentoo and Slackware) encourage (and in the former case, require) users to configure and compile Linux from source instead of relying on a binary installation.
Of course, this doesn't do anything against a malicious install environment or a backdoored compiler, but it's a start.