Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.
In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins.
They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutable deploys, but still just an EC2 instance) then Pulumi+Docker+Fargate so we could fix our deployment velocity. The CTO was constantly afraid everything would break; mostly cause it actually would break all the time. Now basically anyone can deploy even if they're not a SysAdmin.
That's not to say you can't automate a Pet Server, but it's a lot more likely for someone to "just once" make some changes and now you don't trust your automation. In our case we had SaltStack and we were blocked by the CTO from running it unless it was off-hours/weekend.
The "missing magic" I'd like to see is a script that knows I started with Debian, installed these packages, and derives the set up script to reproduce this server including my configuration changes. That way I can make the changes in an environment that works for me and have technology do the boring part.
I wouldn't recommend NixOS on servers. It is unstable and extremely difficult to use. It breaks way too much. I could have done so much more with the time I used to try and make NixOS work for me. The cost of getting it to work is not worth its benefits to me.
When I wanted to set up NixOS, I wanted to use it to manage a bunch of NixOS virtual machines (so, nixops), and then run a couple of services. Here are a few points from my anecdotal experience:
1. Learn an entirely new language that has absolutely horrendous documentation. It used to be (I've been told it got better but I abandoned Nix before this) extremely difficult to debug as well, since oftentimes you would get an error in some library file when the error was obviously in your own file.
2. Try packaging your own stuff. Again, barely any documentation. One time, I wrote a simple Rails application in a day. It took me more than three days to figure out how to deploy it, and that involved just figuring it out. Rails adds a PID file to the directory with your Ruby code in it, but that directory in a Nix derivation (fancy [which makes it harder] name for something like package from my understanding) is immutable. Good luck packaging complicated projects that someone else hasn't packaged yet.
3. Nixops was so incredibly outdated that it caused my server to say it was using an "insecure" library from Python 2.7 and my auto updates started failing. On paper, nixops looked nice (manage NixOS virtual machines/VPSes with Nix). But it broke my updates. I eventually wrote my own replacement for Nixops, which worked, but I still don't trust the rest of NixOS to not break. The tool might have some merit on a different Linux distribution, though. (I'd also note that I believe that nixops fixed their insecurity bug, but there were a myriad of other issues I don't remember that I had with nixops)
4. Setting up pre-packaged services sounds easy. They were, somewhat. But permission errors were pervasive. SystemD tempfiles were finnicky at best and gave path traversal errors that were extremely hard to debug.
Eventually, I got a new server, installed Arch on it, and use docker-compose for everything. It takes me maybe 20 minutes to set up a new service on a good day, instead of four hours of Googling for some obscure error from NixOS. While not perfectly "reproducible," neither was NixOS, because I could simply not trust a virtual machine on nixops to boot successfully—there would always be more required even if it worked once.
I run NixOS, but I recognize that user experiences might be very different depending on the packages you use. NixPkgs is huge, enormous. Some stuff is really well maintained, whereas other stuff isn't.
Furthermore, declarative systems make a difficulty tradeoff. Lots of stuff is much easier, but problems are sometimes harder to debug as there's an extra abstraction layer.
IMHO, these days the best distributions out there are either very simple imperative ones with binaries (Arch, Alpine, etc.) or Nix-like (NixOS and GuixSD). Getting experience in those two extremes is really valuable.
I find people dont know how amazing a "immutable" server fleet is until you've experienced it.
It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour.
It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens of hosts just purring along with 0 downtime since the moment anything became unhealthy, hosts would start auto-booting and terminate the old instance.
"It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour."
Got any tips in this regard?
Ansible, Puppet, Python, Terraform, Openstack Pulumi springs to my mind. Still need to cover a PXE server. (Maybe with Ansible)
We use Salt Masterless with Packer to build AMIs.Terraform defines our infra. No matter what happens we ensure any service can boot by itself with 0 intervention
As turns out a lead developer can't unilaterally change the CTO. Not sure how it works for you. I can control tech, direction, etc. or move on to another job.
I chose to work with the CTO/Team to figure out a solution everyone could live with. I even chose a more annoying solution (Packer) initially just to make sure people felt comfortable and avoid changing things anymore than I had to.
I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists.
Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also.
My hunch is that the 5000 onprem pet servers are not going away any day soon, because a massive amount of it is legacy systems that take a long time to migrate to cloud, if ever. Also the work stress is just ridiculous. So much stuff to do, even with automation. Only reason I still do this is that I like the "old school" tech stack vs. cloud IaaS/PaaS alternatives.
>>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also.
I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g. as you say, we have a room full of linux boxes and we manage them with ansible/scripts/etc, but we design and maintain them ourselves, come join our team".
> When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs
Because the old school systems just chug alongg with a minimum of administration, while those newfangled cloud thingamajigs need tons of adminning...?
I mean, that's one interpretation of your (admittedly, anecdotal -- but that's what I see too. Lots of anecdotes sums up to:) data. A valid one, AFAICS.
You should learn some AWS and use it as a trojan horse to get those jobs. As a former old school sysadmin I really took to it because it's so modular and feels like Unix's "collection of simple tools piped together". Plus, any old school admin is a treasure on any cloud team because the need to drop to shell is unavoidable. People think they don't need sysadmins on cloud teams but behind every good cloud team are a few good sysadmins.
> Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.
You don't need pet servers. Puppet or Ansible make your baremetal cattle.
I think most folks would argue "cattle" means using imaging to manage/replace your fleet. Using something like puppet or ansible against a fresh install implies a level of individualism towards each system as they "may" have minute details based on when puppet/ansible ran, even if they're part of a dynamic inventory of some sort.
There's a single pxeboot server that bootstraps everything. It sends back a single image common among all baremetal servers regardless of what asks for it. The actual configuration is done via a git repo that contains a mapping between mac address of a server and puppet classes.
I disagree. The 'cattle' vs 'pet' distinction is about replicability and replaceability. If you can spin up a new baremetal server quickly and easily in the configuration you need, then you're doing it right. The technology you choose is not relevant.
I've never seen anyone demonize or shame having pet servers, if anything people on tech news sites write about their pet servers constantly (understandably, it's fun!) Just like you're not going to make all your furniture by hand when you start a business but instead just buy whatever works from Ikea, likewise you make a more conscious decision to build or buy (as TFA touched on) based on the constraints of the business. And sometimes a business, for compliance reasons for example, may choose to keep their servers in-house, in which case you could potentially be a sysadmin.
This works fine from the perspective of the admin, but think from the perspective of a business owner hiring admins. What happens when you have employee turnover? You don't want all of the knowledge needed to deploy your critical IT infrastructure stored in one guy's head or personal notebook. Maybe your admins were all disciplined enough to make sure they never made a single change without documenting it and kept extremely detailed, easy to use runbooks, but maybe they didn't. You don't want to get Dennis Nedry'd.
