Isn't it actually worse than that? If some boutique masto admin decides they don't like you or your content, can't they just stomp your account, killing the portability of your social graph, too?

I think of the whole thing as sharding.

Yes, it's entirely possible. The thing is: While yes, the fediverse is "federated", you are still using an account on someone else's server and are entirely dependent on them "playing by the rules".