Even if it's not currently practical for 100 person group chats, verifying each other's keys is quite practical for 1on1 or other small groups of say 10 people or so, and you get a great benefit from doing so! Two normal users could use any matrix server, including the huge matrix.org one, and communicate privately in a direct chat with peace of mind.

Of course, I think self-hosting and decentralizing the servers is very important and good work too!