Without it, suddenly you can hear every conversation happening all the way on the other side of your open plan office. It becomes extremely distracting.
I have bad news for you -- those noise-cancelling headphones are "eliminating" the white noise by bombarding your ears with an equal and opposite white noise
There is nothing bad about that. The opposite phased sounds it plays genuinely do cancel out the vibrations from the original noise, decreasing the magnitude of the vibrations hitting your ear drum.
The HVAC system occasionally stops at my work, in the rare occasion I'm working late in the office, and the silence is profound and divine.
It only lasts a couple of minutes, and it's a frustrating reminder of how fucking loud the baseline office noise is as a result of just the air movement of the HVAC.
It's quite loud, I had assumed it was an improperly installed HVAC system...