Judge John McConnell disagrees with you. His decision placing a hold on the spending freeze cites the Impoundment Control Act as part of his finding that the lawsuit is likely to succeed on the merits.
That decision applies to a very specific item: congressional appropriated federal support to states. Which is probably the strongest case for an impoundment argument.
Yep. And certainly at the beginning of all this there was reason to think that this was happening. For example, Solar for All grants were paused with no explanation of why or the expected path for releasing them.
To be fair, he's a district chief judge and I've certainly disagreed with some of them in the past. But at least it demonstrates that the question of impoundment is on the table.