The Church is meticulous about good record keeping (which is what makes them so good at genealogy in the first place), and every official member of the LDS church has a unique member identification number, used for things like tracking donations, ordinance records, and, of course, genealogy. Nowadays, the vast majority of these records are digitized and you can access them online through Church-owned-and-operated websites. If you happen not to be in the system yet (maybe you were baptized in the jungles of New Guinea and have only just made it back to civilization), it's pretty simple to meet up with a bishop or clerk who can verify your paper records and tell you what your membership number is, which will let you set up an account on Church websites.
So, to summarize, the Church knows who its members are, and knows how to associate them with online identities. When one of them wants free access to Ancestry.com, it's basically the same process as "Sign in with Google" and such- you tell Ancestry that you're LDS, Ancestry asks the Church's servers if you're telling the truth, and the Church's authentication system teels Ancestry to let you in.
You ask the membership clerk of your ward for your membership number, which you use to upgrade your familysearch.org account to an LDS account. Then ancestry.com uses OAuth to get access to your familysearch account.
