Does this always happen or only sometimes? In Google calendar, if the event is modified through the API, there is an option to suppress change notifications, and this can prevent non-google calendars from getting invites.
Also did you create a Google account using your fastmail address? I haven't tested it, but it's possible this causes a hairpin route where Google supresses the outbound email since it thinks it can directly modify the calendar.
It's very reliable, but only on ONE of my fastmail addresses. My Google account predates the fastmail one by a long time.
I had a theory that maybe at some point I set up iCloud with both of these accounts (fastmail + gmail) and it's somehow intercepting these and rewriting them. It's not a super strong theory. Another thought is maybe it's a DNS setting somewhere?
Also did you create a Google account using your fastmail address? I haven't tested it, but it's possible this causes a hairpin route where Google supresses the outbound email since it thinks it can directly modify the calendar.