I think this has something to do with the so-called Parkinson's law, where an organization tends to "corrupt" when it is growing its headcount if its top leader is not aware of it.

I used to work for a startup which has a 'flexible' working time culture too. It was not that bad until the company grew its size to an extent that the middle management layer was established. Then all the shit exactly like what a typical Parkinson's law story happened.

Edit: https://www.economist.com/node/14116121