I assume this is unavoidable. The only solution are protocols where the network is owned and stored in the data (cryptographically) rather than in the servers. Then the servers apply censorship and rules over the data, but you can still rebuild any conversation chain as long as you connect to enough servers that don't censor it instead of requiring 1 server to keep all the network relationships.
This also allows authors to seamlessly switch servers without losing audience or at least being able to recreate it very easily.
That's another problem: Moving your account to a different server in the fediverse. Which is indeed not possible currently.
Perhaps some kind of blockchain would be a solution? (No, I'm not trying to appeal to tech investors, I actually think it might offer just the solution here :P )
nostr (https://github.com/fiatjaf/nostr) seems to be a minimal possible solution. It doesn't seem to be much in use though, so I guess once that happens a few issues will come up.
This also allows authors to seamlessly switch servers without losing audience or at least being able to recreate it very easily.