Google Calendar really has nothing inherently advantageous over every other calendar solution on every other email app. Calendar invites are pretty much a universal standard, and it's perfectly easy to interoperate other people's Google Calendar with your not-Google calendar.
I do the same. I migrated all of my emails over to fastmail a while ago and have been very pleased. There's a web interface, but I mostly use it with the native calendar app on my machine (in my case iCal). A couple of colleagues even switched after I mentioned how much I like it. I've been trying to move all of my stuff off of Google for a while since I don't really approve of their business practices anymore.
Although migrating everything off an old Gmail I used for a decade has been slow. I need to take a weekend to do it at some point.
