Beware the "same config everywhere" approach. It works up to a point, then it turns into a disaster. All you need is one totally broken change like "chmod -x /usr" to really make life interesting. You start bleeding machines and pretty soon you have nowhere left to host your tasks.
It's interesting, right? At first, you can handle a couple of totally mixed-up machines. Then it stops scaling and you have to start doing the whole "golden + syncer" approach.
Then you go too far and get into a monoculture. When the machines do break, it's impossible for humans to go around and fix them in any reasonable amount of time because there are too many. It's amusing when this happens and the solution put forth is "more administrative controls".
Yes, I have a white paper, 'Uniformity Considered Harmful' which addressed some of those issues 'in the large'. The interesting bits are two fold, one is where you draw the lines around the 'meta-computer' and how you approach balancing flexibility about provisioning, around utilization in production. I'll see if I can get blog posting up on this stuff.
You just need to do rolling deployments to make simple things like that go away.
Roll out a deployment to N machines (like say 10), run self checks (you have those, right?), if everything passes give them standard load. Over the next K period of time, periodically check up on them. After that, roll over to N*2 or N^2 nodes, continue until you have rolled out to your entire cluster.
Some overly-confident human went "oh, this can't be bad" and force-approved it. This really happened. Multiple times. And they have rolling deployments.
It's interesting, right? At first, you can handle a couple of totally mixed-up machines. Then it stops scaling and you have to start doing the whole "golden + syncer" approach.
Then you go too far and get into a monoculture. When the machines do break, it's impossible for humans to go around and fix them in any reasonable amount of time because there are too many. It's amusing when this happens and the solution put forth is "more administrative controls".