I have a similar schedule, although a slightly more organized. I try to be available from 9 to 4, atleast through emails if I'm not in my office, because that's when other people also work and might need me. I often leave mid-day to go running or come in late because I went bouldering in the morning. If I don't feel inspired to write or work, I often instead go for a walk and read books tangentially to my research topics. Then some days I'll work 12 hours straight.

I mostly work when it feels right, yes, and do something else when it doesn't. But I am doing my PhD and (in Canada) it absolutely does not pay enough to live comfortably.