I have a suspicion that you might have been able to do it more efficiently (e.g., in one query) using extra() to provide a similar effect to the annotation. That's how I typically handled this sort of thing prior to aggregates landing.