Yes. There have been attempts in the past to insert critical vulnerabilities into open source libraries that would make programs compiled against them vulnerable.

What your suggesting is far less complex than the Stuxnet virus. You need to get binaries from reliable source and verify checksums whenever possible. You can't protect yourself a 100%, but that doesn't mean you should leave or your doors unlocked.