I was a systemd hater. I agree with a lot of the post but in practice I need a whole lot less custom scripts, forking code, etc to maintain a simple service whether it’s a custom c binary, go, rust, nodejs , ruby , python… I don’t need to mess with monit(for most things) and yeah in less words then my post here I could have a working service that would both restart on crash and on reboot…. Do I miss the init.d scripts and writing pid files forking twice to change user … no
No idea why software tries to "daemonize" anyway. Seems to be some kind of weird tradition. It's always been the wrong thing to do. They should just run normally and output to standard streams even if it's connected to a terminal. The reasons are outlined here:
Ironically, systemd is one of few service managers which can properly track those weird "daemonizing" processes. It uses Linux's cgroups. It doesn't matter how many times they fork, they can't hide from systemd.
Not sure what exactly you were pointing to, but section "Doing it right" (4.3.1) recommends:
mydaemon & daemonpid=$!
This is a terrible advice - that's how you end up with all sorts of bad things:
- you ssh'd into server, restarted server, checked it worked. Then you logged out and server got killed.
- your server runs as root or power user
- your server output/error goes somewhere, not clear where
- your server's process got paused by SIGTTOU, because it was trying to write to console
etc...
The complex daemonization dance, with double-forks, closing random FDs, etc.. was required to avoid all this bad stuff before systemd days. Sysvinit/openrc folks came up with "start-stop-daemon" helper, but it was pretty terrible in general, especially in logging department (it had none).
Handling all of those conditions is the job of a proper service manager like systemd. It does things like restart processes that died, redirect output to logs, set process user and group, etc.
The point is software should be written to make the service manager's job as easy as possible by keeping it simple.
I agree that handling all those conditions is the job of a proper service manager.
I disagree that systemd is a proper service manager. Systemd is horribly complicated in comparison with what is actually needed to implement a proper service manager, as demonstrated e.g. by D. J. Bernstein's daemontools and their many derivatives.
Perhaps systemd is better these days, but a few years ago when I have tried systemd on one of my servers I have stumbled upon a bug that could be explained not by some random programmer mistake but only by a bad global architecture of the program. That bug made me mistrust the competence of the systemd creators, so I have never tried to use it again. Moreover, systemd does not provide any feature that I could consider useful and that is not available in alternative service managers, so there were no reasons for further experiments with it.
(That systemd bug was a bug that I have never seen in a computer without systemd, from the mainframes of 50 years ago to the latest computers, regardless of the operating system. Due to a race condition in systemd, sometimes the shutdown or reboot of a computer could fail, with the computer remaining stuck in a deadlock, i.e. in an infinite loop, without properly saving all state and closing all resources.)
Systemd is complicated because software that does it all always is.
You might not need the features, but someone does, and putting it all in one places means everyone can use the same tool, and nobody has to learn 10 different systems for slightly different tasks.
Software doing it all is antithetical to the Unix philosophy of "Do one thing and do it well."
Hence the hate that systemd has, and always will have, as long as there exists a community of people who use Linux because it is a Unix-like operating system.
But, Linux is versatile enough that everyone can have their own little fiefdom. The real rage came in years ago when tools that were widely used by all suddenly became reliant upon systemd to function at all. Thankfully, in the time since, enough development has happened that mostly people are free to choose what they do and don't want to run again. The manner in which it was foisted upon the community, however, was one that hopefully will be remembered as being far more aggressive than should be repeated in the future.
But, I imagine that as long as Lennart Poettering walks the Earth, we likely will have yet another similarly awful episode.
I think a big issue is just that it takes more effort to support multiple init systems. It's hard to be particularly excited about maintaining support for something you don't have a use for yourself, and nobody is paying for(Or at least not as many people in your core audience).
Seems like we're in a pretty good place now, with separate development efforts by and for the people who are actually interested in anything but systemd.
- symlinks - systemd took the "accepted" idea of symlinks from /etc/init.d -> /etc/rc*.d and put symlinks all over the filesystem for all kinds of units. It is less understandable and manageable.
- mediocre config files - they decided that config files are declarative, which could be good, but it pushed simple (and visible) logic in /etc/init.d/* files into more complicated logic encoded in binaries. The simplistic config files seem to be more simple for systemd to implement, but harder to understand, maintain and use.
- binary log files. everything has to go through systemd to access them.
- responsibility creep - systemd started taking over more functions, and it doesn't seem to stop. Will systemd turn into busybox, with simpler poorer implementations of every single thing in the system? dnsmasq is similar. The thing is, busybox and dnsmasq are cut down implementations for smaller system that are alternatives to their full-featured brethren. systemd takes over the core of a system.
> put symlinks all over the filesystem for all kinds of units. It is less understandable and manageable.
It is actually way better understandable and manageable: /usr/lib/systemd is for distro-supplied and packaged units. /etc/systemd is for your local overrides, /run/systemd is for units created at runtime by various generators. (Then there are $XDG_(RUNTIME|CONFIG)_DIR/systemd for user units, for consistency).
So no more diffing config files at update time: the distro provided ones are always updated, your overrides are always preserved, cleanly.
> - binary log files. everything has to go through systemd to access them.
So send them to external syslog, that saves them as you wish. There is a point in indexing them (and being able to have a newline in the log message).
- responsibility creep
This is a matter of perspective. If you take it as "init system at boot", then yes, it might looks so. If you take as an service manager that runs/stops/reloads/monitors units, based on different events (startup, dbus activation, socket activation, timer, etc): what's the difference in the bulk of action, except for the event that caused that action? The service handling, unit definition, etc. are all the same, just the initiation event is different. In this light, it doesn't take over functions; it consolidates and unifies them.
Unlike dnsmasq or busybox, it is not in cut-down fashion, but exactly opposite. It replaces rag-tag of tools with one with more complete and tested implementation.
systemd-network is cut-down I agree. But it does not have an ambition to replace NetworkManager; it is usually installed only in cut-down environments (containers; ubuntu installs it for servers, but not for desktops). Redhat doesn't even have it in repo.
systemd-resolved, on other hand, is good. There's no other resolver stub, that has the functionality that resolved has. It is the only one for Linux, that can do per-link, split DNS properly.
I recall systemd-resolved ignoring the resolv.conf order of preference of name servers. The native and thus relatively simple per-link DNS configuration is great though.
Resolv.conf does not specify order of preference of name servers! All the servers configured here are supposed to return the same answers!
See, many people have misconceptions how it works; see the comments in thread you linked to. Add to it the fact that resolv.conf syntax cannot express what is expected of modern resolvers (not even with apple-like extension - `/etc/resolver/*`; it cannot react to links going up and down), it is basically just compatibility shim for broken applications (they are not supposed to use it directly); that's why systemd-resolved configures it to point to itself and handles the specifics for them.
Providing the same answers does not preclude an order (of preference). The man page clearly shows that such an order exists, though it does not explicitly label it as a preference. Whether you think it's right or wrong is irrelevant—the fact remains that the expectation of querying DNS servers in a specific order has been a common understanding for many. You're free to think systemd-resolved is the right way forward, but that's not an argument against the existence of the ordering in resolv.conf.
> the fact remains that the expectation of querying DNS servers in a specific order has been a common understanding for many.
That understanding has been wrong; it is an implementation detail of glibc and that's it. It could query them in random order, or round-robin them for each request; or time them how fast they respond and prefer the fastest, up until it starts timing out and then shake the order again. Some resolver stubs ask all of them and use the answer that arrives first. That's also valid. Since they all are returning the same answer, it doesn't matter. It also makes any argument for ordering superfluous.
So that "common understanding for many" is erroneous and those many should update it. The thread you linked to has exactly a discussion on this point, so in the years that passed since then, those affected many had ample time to fix up their assumptions.
That understanding is in line with using libresolv and the established behavior of the resolv.conf file, which accompanies the libresolv library. While it’s true that different implementations (e.g., some resolver stubs) might use various methods like round-robin, timing, or querying all servers simultaneously, this doesn’t invalidate the behavior outlined in resolv.conf.
Again, there can be other valid reasons for maintaining an order. Could be about latency, privacy etc.
I admit to not having read the full thread, and I apologize for any needless rehashing of arguments. It was just the first result confirming systemd-resolved breaking setups by its handling of what many (in that thread) consider expected behavior.
What I think you're primarily suggesting is that there are no formal standards prescribing glibc's behavior in terms of DNS resolution order. That is certainly true, and I don't disagree calling it an implementation detail.
I think it's possible that you just hold a non-majority opinion about what good usability looks like.
That's fine, and I'm OK with a world where we disagree about how usable systemd is, but I interact with the systemd config files and symlinks and logs pretty frequently, and have found them to be orders of magnitude more usable and understandable than /etc/rc#.d and /etc/init.d and "I wonder if this system puts sshd logs in /var/log/auth.log or /var/log/secure or somewhere else". Notably, I interact with a mix of Arch and Ubuntu and RHEL systems over the course of a week, and systemd means I can engage with all of the above using the same methodology.
In terms of creep... busybox is noteworthy for packing all of a system into a single binary. systemd doesn't do this. Their resolver and time daemon and networking and all the other bits and bobs are their own separate binary, and you aren't required to use them.
It is possible I haven't communicated the gist of the idea.
systemd is functional. It does the things you say. It unifies things, including config files and log files. It works for you.
I just don't think the way it did this is that elegant.
My busybox analogy should be more viewed as: what if arch/ubuntu/rh decided that busybox would replace all coreutils, etc on their systems today? All the same commands. Still usable. But something is missing...
1. symlinks: I think the systemd structure is very intuitive and the order of overwriting makes sense. It's new, but it makes sense, and we only have 3 different locations, each with a specific order and precedence.
2. Config files: I think keeping the declarative syntax is the secret sauce of systemd. The whole point is you can orchestrate processes without needing to write code. They're much easier to understand if you assume a knowledge point of 0, and they're much more maintainable (no logic), and they're even more portable. Bonus is they're much faster to execute by the init system. Yes, most of the logic is in systemd now, but you tell it what to do. Reading the documentation goes a long way to making unit files less cryptic.
3. Binary logs: yeah I agree just not a great idea. Always sounds nice on paper because you can do neat things with permissions and can have faster performance, but in reality just not necessary. They got too fancy.
4. Responsibility creep: from where I'm standing, not a problem. systemd isn't a piece of software, it's just a name given to a collection of software, like GNU. systemd-init doesn't have feature creep. Sure, now we have systemd-file-my-taxes, but that's optional and it's a whole different program that just uses the systemd name.
Unfortunately, there aren't any concrete examples (unless I skipped something accidentally), so it's hard to tell what exactly is meant as "does more" or that it doesn't "stand out of the way" etc.
Yes, systemd services can do a lot of things. Except that before rc scripts did all those things, but without any unified and standardized primitives. systemd provides those for a lot of common use cases - and that's how it ended up doing so much - because our rc scripts used to do that. systemd actually does less because unit files are not scripts (but one can hook any external program in a lot of places, if they need to).
Also, as I understand it, systemd isn't a single piece of software either, it's a whole project with a bunch of semi-autonomous pieces, tied with common conventions and assumptions. It's difficult to define the scope for an umbrella project, and I believe individual pieces all have their scopes relatively well-defined.
>> The single, overarching problem with systemd is that it attempts, in every possible way, to do more instead of less.
The more systemd does, the better I like the power, features, consistency and functionality my linux systems have.
systemd is a beautiful work of art.
Software developers owe it to themselves to full understand what systemd has to offer, because if you really understand it then you'll find you can architect your systems around systemd and often let it to the hard work for you.
If anything, the old gnu/linux name should be dropped and replaced with linux/systemd to illustrate that in fact systemd is really a major part of what Linux is.
This reminds me of teenagers of the last generations who preferred distorted mp3 sound to original because that's the sound they have grew up with. Teenagers of today prefer distorted wide-angle selfies instead of properly shot faces because again that's what they have grew up with.
Calling systemd a work of art gives me the same vibe.
I can't tell if it's sarcasm or genuine, but I 90% agree. There's a lot of things that it makes easier. But there are areas which are just weird and overengineered and overcomplicated - for example resolved interacting with old files and creating lots of possible ways it can run. If you look at forums / SO it's obvious people are confused about what it tries to achieve. Instead of creating a whole new system, they could've started by implementing and standardizing the multi-interface behaviour in established projects like dnsmasq. Instead of that they went with NIH where it really wasn't necessary.
For resolve.conf, I think it is fundamentally bad design that is hard to get right. Before systemd, I've spent plenty of time trying to get networkmanager and resolvconf to play together. And then there was those dhcp scripts which would edit the files directly.
Also, what do you mean my "standardizing dnsmasq behaviour?" From my recollection of using it (and from manpage[0]), it's pretty basic. You (1) manually edit /etc/resolve.conf to point to localhost dnsmasq instance (2) use app-specific ways to configure nameservers, in dnsmasq's case it's an ad-hoc list of secondary resolv.conf files specified on command line, with no defaults. I don't think this is worth standardizing, as this scheme cannot even represent network interface -> dns entry associations.
I didn't say standardizing dnsmasq behaviour. I said multi home/interface which was one of the big reasons to write resolved. They could've added the missing feature to the 2 most popular daemons and use one of them as a default, but instead went with a whole new system written from scratch.
> If you look at forums / SO it's obvious people are confused about what it tries to achieve.
Well, many people are confused about how DNS resolving works. Many of do not consider a problem, when they configure their resolv.conf to two different upstreams, each returning different answers (i.e. one responding for domains that the other doesn't) and then wondering, why stuff is broken.
Of course that's a bit of a troll comment at the end there. There are plenty of Linux distros still not using systemd. https://www.devuan.org/ as one example.
Trying to compare Windows updates favorably to updates on systemd seems a bit bizarre. I can't ever remember seeing an upgrade ask for a reboot on Raspberry PI OS (debian 12). And I can't ever remember seeing a Windows update that didn't reboot the system.
As a long-time Windows developer (both apps and device drivers) substantially coming to Linux long after the decision to use systemd was made, I am impressed as heck by Systemd.
And was completely horrified by my very limited experience, as a user, with pre-systemd Linuxes. That was insane. Manually edit what in to which system file? And my system invariably didn't even HAVE that file.
Even writing systemd units is virtually effortless.
I get frustrated when people misstate the Unix philosophy.
The Unix philosophy is about function, not form. Say you have a tool to send something over a network. In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
It's a consistent insistence that the 'single job' be defined today just as it was in 1980 that results in Linux requiring hundreds of packages to be even remotely usable. Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
It is currently the only linux resolver stub, that supports link specific upstream resolvers. In the terms that user would understand, your split-horizon DNS works correctly over your VPN.
> Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
WiFi absolutely should be a separate component; the unix philosophy is an excellent defense against servers in a datacenter trying to install a WiFi stack.
wifi happens to be a separate component; wpa_supplicant or iwl are separate user space tools that network management tools call into; on the kernel side, the drivers and their stack are separate modules.
Yes, as it should be; while we might quibble about the exact details, I think the Linux-world network stack(s) do a good job of separating components cleanly while still giving users a good experience (if you just install Debian on your laptop, you don't need don't need to know or care how many packages it takes to make networking work, it just does).
The Unix philosophy is about function, not form. Say you have a tool to send
something over a network. In 1993, that tool can get by being quite simple.
In 2024, that same tool now needs to be more complex, just to do the same
'one job' and to 'do it well'.
Even by that standard scope remains a problem with systemd, it makes emacs look svelte. It's absorbed big pieces like time management and logging that ought not be part of a purported init system.
A core part of the Unix philosophy is being able to pipe standard streams between processes, files, sockets, etc, as the user sees fit.
Systemd, particularly journald, violates this severely by snarfing any stream you want to send to /var/log or wherever and locking it in a binary database somewhere, that you have to use journalctl to access.
I have no problem with journald as an idea, but the audacity that it knows better than the user what to do with stdout/stderr is insane, and 100% anti-Unix.
> In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
1993's sendmail is more complex than qmail. The latter uses a modular system, where each subtool does its job well, rather than sendmail's monolith design.
Linux was(is?) the system. Shell scripts could configure the system, reasonably well. Systemd meant it was no longer possible to simply fix a linux distro that had it...a massive recompile and intricate understanding of many low level systems was needed.
No, you shouldn't need to know about PDP-11s to fix your buggy wifi, neither should you need to know a massive binary subsystem to configure your wifi either...
So as far as doing its "one job", it did/does it fairly poorly, even today. It is a lot more standard and it does get points from me in that respect, but it fails at being a configurable system completely.
I could simply trace shell scripts with some file io, or echo statements. Now there are systems using python, even better I can set breakpoints and modify, no need to recompile.
Configuring the system with textfiles only works if the system anticipated all your needs, and to anticipate all needs is not only complex but error prone.
There have been times where I needed to do something that wasn’t available in the unit file syntax, so I just do it in a shell script and ExecStart that.
I don't think systemd is meant to be fixed or modified, unless you are actually contributing to the upstream systemd project and have accepted that it will be a time consuming project.
It seems to be made for, and works well for, a modern ephemeral infrastructure workflow.
Build stuff to work with an unmodified distro, wipe it and reinstall if it goes wrong.
I was indifferent to systemd until podman shipped quadlet. Now being able to bring up containers, tied to the system, in a reliable and efficient manner, without docker compose or big bespoke commands, has been incredible
Every time systemd comes up everyone says it's great and how much better it is than SysVinit. No one EVER talks about the actual alternatives that people use like OpenRC, runit, s6.
By all means, be the change you want to see. All I know about OpenRC is that it doesn't support user services the same way systemd does. I know even less about runit or s6, so I'd love to hear more from people who use them in their daily computing. Because unfortunately the impression I get from never seeing anyone talk about OpenRC, Shepherd, Upstart, et cetera; is that either nobody uses them, or they don't add anything worth noting over SysVinit.
Though I'm surprised that you say:
> Every time systemd comes up everyone says it's great and how much better it is
When my experience is usually that people complain about systemd and it being over-engineered and doing too much and having too many components (yes, I know that those projects are related to systemd, not part of it, I'm just saying what I see people mention as downsides).
OpenRC just works for me (Alpine) there’s a declarative side that’s more than enough for simple services, but you can override logic for starting, stopping, reloading,… as it’s just a shell script.
> unfortunately the impression I get from never seeing anyone talk about OpenRC, Shepherd, Upstart, et cetera; is that either nobody uses them
People do use them. Multiple distros either default to one of those, or support it: Alpine, Devuan, MX Linux, Slackware, Void, Guix, Gentoo and others.
> or they don't add anything worth noting over SysVinit.
Not looked at this myself, but do they mostly not add something - dependencies, easier config or something?
systemd is bringing Linux feature parity to Windows NT in the mid-90s. That's 30 years ago for those of you keeping score at home.
Funny the author mentions Solaris because Solaris has SMF and other systemd-similar tools.
Linux pre-systemd looked like a kernel duct taped to a bunch of random thrown-together shell scripts... because it was. A sorry state of affairs that a bunch of very stubborn people were afraid to let go.
The simple fact that configuring networking out of the box is completely different on every distro and sometimes within distros depending on which tools were loaded, in 2024, is simply insane.
That's what makes Linux great. The entire Linux user space is replaceable. You can trash it all and make your own if you want to. You can trash systemd, write your own program and boot Linux directly into it if that's what you want. And people do, myself included. Not a single other operating system lets you do this. There's always some component that you cannot touch.
I'd rather have the chaos of Linux distributions than the tyranny of "standard" ways of doing things.
> You can trash it all and make your own if you want to.
It is important to note, that if you want to do it, you have to roll up your sleeves and do it. Not to expect free manpower of others to provide it to you.
The biggest reason systemd is taking over is companies like Red Hat keep paying people to work on it. As time passes, systemd gets better, gets more features, makes the life of distribution maintainers easier and therefore gets adopted, and makes the lives of users easier too, mine included.
I'm making my own freestanding Linux thingy but I'm not under any illusions that it's going to replace systemd or any other software any time soon. I'm just one guy doing it in his free time because he enjoys it.
I really hate it when I see people complaining about systemd. The people behind systemd had a vision, they reimagined Linux user space and put the necessary work into making their vision a reality. They're an inspiration for me. Not only that, they took their work and they released it as GPL licensed free software. One would think the users would be more thankful. One would think people would not have moral objections to free software getting released to the world. Some technical criticism is certainly warranted but the sheer hatred that people display towards it and its developers certainly is not.
I agree that "Linux is about choice" is often misinterpreted. It's about my choice as a programmer and software developer to redefine the entire system downstream of the kernel however I want. Users are encouraged to themselves become technical and educated so that they too can enjoy this power. However, we're not about to drop everything and start maintaining some legacy system because people who aren't paying us keep insisting that it's how things gotta be done. We're programmers, we like it when things are made in our image.
Configuring networking is different not only between distros but also between different versions of a single distro, say, Ubuntu. Every major release lately brings new networking configuration method while abandoning the previous. Is it frustrating? You bet. Thankfully I can still `apt install ifupdown` and it works, but who knows for how long? systemd was optional for a couple of releases as well (`apt install upstart`) but of course not anymore.
Funnily enough, the network configuration problem is not due to systemd -- which has nothing to do with it -- but with Canonical being Canonical. Someone here has a new toy (netplan), so why not force it on users, with some excuse.
For the same reason, why Canonical forces netplan (consistency between desktop, using NetworkManager and server, using systemd-network), Redhat standardized on NetworkManager instead.
Sure, it has nothing to do with systemd. The vibes are extremely similar though. "We don't care what you think we are kids with a new toy and we will shove it down your throat no matter what" kind of thing.
(I hate NetworkManager just like I hate netplan though. Both can get off my lawn.)
I have to wonder if this is some kind of statistical bias based on the type of engagement.
There are some people who are vehemently opposed to systemd, for a variety of reasons with varying degrees of merit. I think it's fair to say that systemd's developers broadly ignored those people, figuring out (correctly) that if they convinced a majority of distribution maintainers, and made the user experience for the average user smooth enough, that was actually going to be successful. As it should be: distro maintainers get to pick how their distros work, and users determine which distros they use.
And systemd has a clear sell for those groups: distro maintainers can standardize their service scripts, often just using what an upstream package author can write agnostically to distro, and users can handle the majority of base system functionality without having to pick things like "which of these cron daemons should I install".
I'd also add that systemd appeals to application developers (no daemon code, one file for all distros). And it appeals to sysadmins too (no more /etc conflicts on any package update! Modify any system service! etc..)
Which brings me to the question of which category systemd does not appeal to? I am guessing "minimalists" and "people who like things done the old way" - and those are simply not that numerous compared to others.
> pick things like "which of these cron daemons should I install".
And we're back at it: systemd seemingly has plans to eliminate cron. I was not impressed when I have been pulling my hair out on why certain jobs in Ubuntu still run despite being removed from cron files. Turns out, systemd now has 'timers'.
So, mark my words: a few Ubuntu releases down the line, cron will become an optional package and after a couple more it won't even be available.
Many parts of systemd were available for long time but fortunately were not used in distros. And then out of the blue Canonical decided that socket activation for sshd is the way.
Yes. Systemd timers are an order of magnitude better for the average distro maintainer and the average user than the various ways of invoking and running cron.
Surely for each individual distro, they get to make their own decision about how to build their system.
I can’t stop you from hating, but if your pitch is that distros shouldn’t be allowed to make their own choices about what to include if you disagree… I guess your option is to start your own distro.
No linux distro was forced to adopt systemd. They did it on the merits, from their determination.
I have to strongly disagree on the last one. Systemd is pretty much a canonical (pun intended) example of a solution in search of a problem.
Systemd as PID1 is nice on merits. I wouldn't want to get back to sysv initscripts.
Everything else is incredibly hostile to practical unix philosophy and solely exists out of spite and politics. This is my opinion and I realize I'm in the minority.
How does that account for the distros using systemd-networkd, timesyncd, resolved, etc, where they’re separate binaries and services, instead of their predecessors?
If the other binaries aren’t better for maintainers and their users, why are they replacing other tools with them?
I think it's change for the sake of change. I think it's mostly politics coming down from Red Hat. How exactly I'm not sure, but I fail to see any sane reason behind replacing things that worked well for decades with new things that are oftentimes inferior on merits.
Anyway. systemd vs unix philosophy is an insurmountable ravine and I chose to stay in what I think is sane camp.
I think maybe it's worth considering the possibility that the things being replaced did not work well, and were replaced because the new options solved problems and user pain.
To pick on cron, just from my own experience:
"I wonder where this scheduled task is configured" -> "Lemme check /etc/crontab" -> "nope, I guess maybe /etc/cron.d?" -> "Nope, lemme check /etc/cron.hourly/daily/weekly" -> "hmm, not there... maybe I want `crontab -l`... or it's running as a user, lemme check that list and run through `crontab -l -u $USER"
Finding out that I didn't notice a cronjob was failing because the stock methodology for notifications is email, and relies on the system having SMTP configured and having MAILTO set in cron.
Realizing a cronjob wasn't running because some, but not all, of the cron options have character limitations like "the file name can't contain any dots".
The total lack of dependency management, so anything I wanted to run @reboot needed to handle its own dependency checks and sleeps to make sure I didn't say, try to run puppet before the network was up.
On the networking side, I started using Arch when the config flow was "edit bash arrays in /etc/rc.local". I've used netcfg, netctl, and now systemd-networkd. systemd-networkd is easier to reason about, easier to debug, more powerful, and more portable between systems.
As much as I don't want to, I have to admit you have merit here on all points.
My question though: how did we manage to run infrastructure using cron for decades and did not feel any of that pain?
My take on the answer is this. Failures: we have /var/log/something for this and typically it's obvious something is not being executed. Many files: yes, there are different configs but don't you remember where you typically put your cronjobs? For example I always use user's crontab, so this is where I would look for the job spec.
Dependency check is one of the main reasons for systemd as PID1 if not the main, so hands down, that's valid and true.
We did feel pain. You could mitigate the pain by never making mistakes (like by ensuring all your cron scripts log to somewhere in /var/log, or that you have alerting, or that you always put them in crontab).
But also what we consider to be painful has evolved. When all systems were to some extent a hand-gardened entity, maintained by some specific system administrator, it was easier for them to have a personal standard. It didn't matter if I used cron.d and you used crontab, as long as on our own systems we kept one method. Increasingly, servers are more ephemeral, more horizontally scaled, and more automated, and systemd timers are just easier to work with in that model.
In a decade or so, I'm sure we'll have further modified how we handle systems, and something else will come along that solves painful things about systemd.
Related, it's also blatantly ignoring that it's the first software that even allows interoperability in its space, and was thus in huge demand by both developers and administrators.
(Edit: Other anti-systemd rant sites sometimes mention these obliquely as "upstream dependencies" and "workplace-mandated", while ignoring the fact that those people have reasons to demand those things)
But HN really doesn't need to litigate this all again.
Conformity is not interoperability. If anything, systemd is like Chrome, so popular that it persuades people to drop interoperability outright and just run with its nonstandard APIs.
Upstart and OpenRC both came years prior. As a matter of course anything which was self contained and standardized could have allowed interoperability. Systemd just actually convinced everyone to standardize not least because its integration with other parts of systemd provided both a carrot: standardization and simplicity for distro maintainers and a stick: the extra work required to use only part of it or especially if one used none of it and was expected to both write ones one unit files AND integration with stuff like gnome which was and remains the most popular singular GUI.
I (and this applies both with my dev hat and with my admin hat) can now write a unit file, depending on a different unit file, that works the same across all distros (note that this includes random end users whose systems I don't directly have access to). This even works for services that start per-user; no more write-only modifications to user startup files or DE-specific autostart dirs.
Pre-systemd this was unthinkable.
(similar stories apply to non-systemd-proper parts of the systemd project, but I personally haven't used them as much)
in the bad old times, every linux distro implemented its init scripts in a slightly different way, often with distro specific tooling and commands for accomplishing most management tasks
systemd unit files require minimal modification by distro repackagers.
With SysV-init, it was/is necessary to modify /etc/init.d/servicenamed shell scripts to get consistent logging and respawning behavior. With SysV init it's not possible to add edges between things to say that networkd must be started before apached; there are 6 numeric runlevels and filenames like S86apache, which you then can't diff or debsums against because the package default is S80apache and Apache starts with A which alphabetically sorts before the N in networkd.
And then add cgroups and namespaces to everything in /etc/init.d please
It's possible to pipe stdout to syslog through logger by modifying the exec args. But I'd rather take journald's word for it.
Supervisord (and supervisorctl) are like s6. I don't think any of these support a config file syntax for namespaces, cgroups, seccomp, or SyscallFilter. Though it's possible to use process isolation with any init system by modifying the init script or config file to call the process with systemd-nspawn (which is like chroot, which most SysV-init scripts also fail to do at all, though a process can drop privs and chroot at init regardless of init system).
systemd works in containers now without weird setup. Though is it better to run a 'sidecar' container (for e.g. certbot) than to run multiple processes in a container?
> It's basically an integrated redesign of all the low-level userspace of a Linux system
Yes, and that's okay. Whatever it is that came before it sucked so much I don't even remember what life was like before systemd.
Also, it's amazing that we even get to redesign Linux user space like that. That's something special. Not a single other operating system allows you to do this. You could boot Linux straight into your program if you want, without any user space at all. Linux is such a solid foundation it lets you swap out the entire user space, you can trash systemd and even the entire GNU stuff and just do your own thing. It's inspired me to start working on a redesign of my own.
Yes. However it's not at all obvious that there is a clean separation between Linux and user space.
You look up the manual entry on some Linux system call and it gives you the glibc documentation. You look up the init system in the manual, expecting details on what the kernel requires of PID 1, and it just gives you systemd documentation. This stuff is completely mixed up.
You look Linux up on Wikipedia and the article eventually starts telling you about standard C libraries as if they were part of Linux. In some articles there's even a diagram showing glibc wrapping the kernel, as if it was Linux's equivalent to the Win32 API.
If you haven’t seen this talk[0], I think it covers in a fair shake the positives, negatives, and what even a post-systemd world would have to deal with.
It wouldn't surprise me if Benno Rice wrote that talk in response to this exact blog post – though there were enough posts in this same genre at the time, and he could have been responding to any/all of them.
Benno's entire argument is that Linux is the only operating system that matters and whatever problems systemd has are irrelevant. Everything else is obsolete. Especially when couched as "look at what this BSD guy thinks of systemd" it's a disingenuous argument. That talk is no more a fair shake than anything to come from Lennart's mouth.
Look, I use linux, OpenBSD, Apple and even Microsoft operating systems. I thought the video provided a unique perspective to the systemd discourse that I thought people would be interested in seeing if they hadn’t already.
I couldn’t care less if Debian announced that Debian 13 is going to default to s6 or runit as the init system. I have no loyalty to systemd other than the Linux distros I use have integrated with it.
The linked article introduced me to s6 which is interesting. I do think there is value in simpler codebases a la wireguard, toybox, and doas. None of the hostilities towards one init project over the other have ever made sense to me but “the battle” was mostly fought before I got into Linux.
What's unique about it? That Benno tosses out a reference to BSD here and there? Otherwise it's just a video of someone dismissing criticism out of hand. Could've gotten that from Lennart.
The author also makes it sound like something has been taken away from them - one can rip it out if they really want to. It's just that most of us have our focuses elsewhere and have no need to tinker with systemd.
My issue with the "systemd violates the unix philosophy" argument is that it... doesn't. I see systemd-resolved touted as an example of this. systemd-resolved is a different tool to systemd; it does one thing and does it well.
Might as well say GNU violates unix philosophy because it has text processors AND a shell.
Kind of a non-sequitur, but what is actually the point of systemd (and other similar tools)? Like compared to just spawning a bash processes that run a critical commands in a loop.
I can come up with a couple functions, but none of them seem like core things, more like tangential benefits:
- Startup ordering means you don't get log spam with "can't connect to X" until X starts
- Certain dependencies like mounts look the same whether they're "running" or not, so not starting before the mount is important. Or like, if a program enumerates interfaces or something, making sure all the interfaces exist first (via a dependency). Of course, a program could do its own checks and possibly be more stateless about it.
- As a standard, for packaging - you can distribute a service/socket file with your program and most distros can just plug it in and expect it to work (also thanks to having standard targets and directories).
Otherwise it seems like an ugly declarative DSL for gluing programs together, which you could do in bash or python or whatever other language too. Is reducing log spam really the point? FWIW I do think systemd is better than the alternatives ATM...
On any system, something has to start "first", and be in charge of figuring out what the heck happens next. On modern Linux systems (and I'm oversimplifying massively), the hardware hands off to the kernel which launches some kind of init system the first userspace thing. That could be sysvinit or upstart or s6 or systemd or any number of other things. You could write it in any language, though the nature of when it has to launch strongly incentivizes writing it in some kind of compiled language so you aren't having to make sure the right interpreter bits are in place first.
That thing then, depending on its level of complexity, gets to handle things like "what else do I launch" and "what order to I launch it in" and "what do I do if a service I started stops on its own".
Beyond that, your life is way easier if more than one distro uses the same thing to do this, so that an upstream package author can write "this is how an init system launches my service" and distro maintainers can use that, vs having to create and maintain their own scripts for how their init system launches a service.
So, the "point" of init systems is "something has to be in charge of userspace".
Sorry, I guess my question wasn't "why does the tree need a root process" but "what is the critical feature that makes it worth using these tools instead of the dumbest possible alternative?"
So the dumb response to your questions is if I have a list of services: "what else do I launch?" everything, "what order do I launch it in?" launch it all at the same time, "what do I do if a service I started stops on its own?" restart it. I typoed, but I really meant my first sentence verbatim: suppose init is a bash script that just does:
On a desktop I'm not starting/stopping systemd services ever. Every process I launch is managed by the window manager. On servers it's mostly the same: start everything at boot, then the server does its thing until shutdown. On more traditional mutable SMB shared servers I can imagine sysadmins going in and stopping/starting/adding new systems, but I feel like that use case is gradually growing less important.
It's largely touted that systemd's killer feature is that it represents services as a graph. But why is it important that it operates as a graph? Why is ordering startup important? Or does that not matter, and the real reason systemd matters is that it makes packaging easier (as you said, and as I said) and it has lots of built in conveniences? I have my guesses, but I'd like outside opinions.
This very much reminds me of how old Slackware systems used to operate. It was workable, but had a number of problems:
(1) I want to stop "service1" for a few minutes, without affecting other services, how do I do this?
(2) I want to disable "service1" completely, or pass it an extra parameter, without affecting other services. How do I do this? Note you cannot edit the init file and restart, this'll restart all the other services as well.
(3) I just typed "my-package-manager install service4", and I want service4 to start right away (and naturally have full functionality - keep restarting on failure, save logs, etc..). How do I do this?
(4) service2 had a emitted too many log lines, and my log disk is full. How do I clean existing logs without restarting the service? (Note even if you remove the file, it'll still take up the disk space)
(5) You were debugging service3 performance, and noticed it has printed "ERROR out of foobars" to its log but didn't print the timestamp.. When did this happen?
(6) service2 needs to be started after NFS volume is mounted. service3 is responsible for cpu fans and should be started as early as possible. service1 will re-generate the userdb if the data directory is missing, so it should not be started automatically if that's the case.
(7) new version of service1 changes command-line and default restart interval. How do I apply this change to my heavily customized startup file?
I could go on and on. All of those are real problems, and before systemd, they were solved with tons of bespoke code. Either C/C++ code (making a trivial daemons complex beasts), or shell code of init system, with was typically highly distribution dependent. And of course it was fragile and full of bugs.
Thanks! This is exactly the sort of stuff I was looking for.
So maybe this is too reductive, but basically:
- Tools for mutation of a running system
- Dealing with hard dependencies that don't cause errors when unavailable (like nfs in your example, or fan control, but also enumerating devices as I mentioned earlier, or for security, firewall rules since firewall starts entirely permissive)
- Standard packaging (edit: er, composability via standard packaging, or modularity with standard packaging for modules)
I think the logging stuff (datetimes, clearing, etc) is probably more the domain of a logger like journald, whose purpose seems is more concrete to me.
This is pretty fundamental stuff. You can't run a program if the disk that it's sitting on hasn't been made accessible yet. You can't start a network service that listens on a certain IP address if that IP address hasn't been configured yet, or if the destination for service logs isn't ready, or if our service needs to talk to some other service like a database which isn't running yet. Etc etc. Booting up a modern system to a graphical environment requires hundreds of things to happen in the right order.
As for everything else, the project announcement blog post from 2010 does a pretty great job summing up the shortcomings of doing "the simplest possible thing": https://0pointer.de/blog/projects/systemd.html . You can certainly still do things that way if you want, but most people value the benefits gained from a more intricate approach.
Ah, thanks for the link! So it has an interesting point, which is socket/device activation. And services that aren't started immediately could also have dependencies you don't want to start immediately.
I can see that being important for a desktop, where you're plugging/unplugging stuff and want startup to be fast.
> You can certainly still do things that way if you want
Sorry, I didn't mean you specifically. I meant, if someone values simplicity above all else, they can choose an init that matches those values (like the one you described). Your approach is perfectly valid given that set of values.
Debian and other distros asked their userbase which values to prioritize, and the users chose what systemd provides. Hope that helps explain "why does systemd exist / why have most distros chosen it?"
Oh, that answer is bigger but pretty boring: the dumbest possible alternative doesn't account for the complexity of real life systems.
Some examples:
On your desktop, while you launch programs through the window manager, things like the window manager itself, and also the network daemon, and time-keeping daemon, and the daemon that logs go to, etc etc, all need to be spun up by something.
Some of those daemons should restart themselves if they crash, and some of them shouldn't.
Some of those daemons misbehave, so it's nice to be able to tell them "don't eat all the RAM" that rendaw is trying to use to do whatever you're using the desktop for.
Some of those services shouldn't start until it's the right time in the sequence, but also you want booting to be fast, so where we can be parallel, it's nice to be parallel (this is where the graph comes in). And on the flip side, some of those services need to restart if a service they depend on restarts.
I've run s6 (the project built by skarnet, the author of the original post here) on a variety of systems. I would highly recommend it as a very high quality example of the kind of minimalist init system you're envisioning. And likewise, I'd encourage you to try it out! It's possible you'd find that for some/all of your use cases, the value of simplicity outweighs the value of the conveniences. But the reason that systemd has seen so much success on modern Linux distros is that for the average distro and average user, the conveniences in terms of functionality and standardization are worth it.
Okay, I assumed there are services that shouldn't restart if they crash given that that's the systemd default, but do you have an example? I don't think I've ever encountered any, and I can't think of a situation where you'd want that.
If you have access to a systemd-based system, you can try:
grep -R Restart /etc/systemd/ /usr/lib/systemd/
And examine the returned unit files. But one of the examples I see is things like the SSH server (useful so you don't lose management access to a remote server), the virtual terminal process for local console logins, etc.
One of the examples from my world is generally databases. If I’m running a large or clustered DB, I probably don’t want it to just blindly restart on a crash.
I hate systemd. I sincerely and strongly believe it to be the worst thing that happened to the world of Linux. Note: this is just my opinion, of course I'm obviously wrong, can I have it please?
Having said that I have to admit that the battle has been lost and there are no practically suitable server distributions clear from it, let alone desktop. So the only fight worth fighting is to delay destruction of other parts of the OS. In Ubuntu you can easily remove systemd-resolved, get rid of ssh socket activation, bring back cronjobs to crond and install a sane NTP client. Hopefully, we will still have sudo available for many more years at least as an optional package and then as a third-party deb. And thankfully we still have logs in /var/log and journald is not a hard requirement for anything. Yet.
Otherwise resistance is futile as systemd people excel at politics and seemingly no opinion or input has any influence their decision making process.
Again: I'm totally wrong here and systemd is good.
Systemd being one big monolithic system locks you in to, frankly, just a lot of shitty decisions. Take for example the fact that it decides to bind to port 53 on localhost, for absolutely no good reason. Now if your response is that is not a big deal, or you push your glasses up on your nose and want to explain why its important, imagine for a second if it was port 80 or 443.
I have no real complaints about systemd as an init system. I have plenty to say about systemd not staying in its lane.
systemd's developers have formed a habit of deciding to implement something like an NTP client or DNS resolver, bringing a healthy dose of ignorance of how those services actually work, and then doubling down on their poor design decisions when problems are brought to their attention.
But that's systemd the project, not systemd the init system. You don't need to use any of those components to use systemd, which has no dependencies on them. Those services can live and die on their own merits.
Which should make you consider its validity. If all these distros chose it, independent of one another, there's gotta be a reason. The reason is that it's much, much simpler and more maintainable from a distro perspective and a sysadmin perspective.
Ultimately distros decide what software is on their distro and users decide what distros to use. So the solution here is to find a distro that meets your needs.
The systemd monorepo also provides systemd-resolvd which does, but this is optional and last I checked not included or at least started by default on Debian, for example.
If only it was that easy. Unfortunately, systemd-resolved is a sticky piece. You have to `apt-get uninstall` it, then `rm /etc/resolv.conf` and replace it with a sane file (`nameserver 1.1.1.1` or whatever).
For previous versions I had to `chattr +x` as well because systemd stuff of course will not let me have my resolver.
Indeed. It has to be completely removed from the system and even then there is no guarantee it won't be brought back by some minor update in a couple of days. This is why I had to chattr +x in Ubuntu 22 and older. Not anymore in 24, but we'll see.
Wrong and wrong. It just needs to be stopped. And you can mask it to make sure it never ever ever is started by anyone again. C'mon man, 0 for 0 so far on your grumbles.
Do you know how to use netstat to see who is holding the port open? This seems like a major Problem Exists Between Keyboard And Chair issue. But also Ubuntu can be a special place. But if you stop the service & the port is still open... It's probably not that service, it's something else, and so throwing shade at systemd is again not accurate.
Ah. It might be a systemd-resolved.socket (or similarly named), allow socket activation. Systemd would warn you when you disable the service that it wouldn't be giving up the socket, that it could still be restarted. Because it's a great monorepo that's smart & courteous.
I think we're confusing two things here. ssh via socket activation is of course a brain damage done solely by Ubuntu. But systemd goes a long way to make sure it hurts, because `ps ax | grep sshd` will definitely show you that /usr/bin/sshd is running. Except it's not it. But maybe that's Ubuntu thing as well, not sure here.
You would know to run netstat or lsof if you have expected something like that to happen. It certainly wouldn't ever cross my mind to have sshd socket-activated and systemd actively trying to hide the fact. This change by Canonical was out of the blue, totally unexpected.
You can have default one. Chances are, if you have only one configured, it will be default. How are you forced? Who did that? How?
Oth, with resolv.conf syntax, link-specific resolvers are impossible. Even the apple-like extension in /etc/resolver, they cannot respond to link doing up and down.
In NetworkManager, you set your DNS servers using `ipv{4,6}.dns`. Similarly, you set `ipv{4,6}.dns-search` with zones that you want to be looked up via DNS over this link. By default, the default DNS is the link with with default route. If you want to designate the default DNS via different link, set up the search for `~.`
Then `resolvectl status` will show you each link, what DNS is configured for it, what domains it will resolve and which one is the default.
I was struggling with it while trying to troubleshoot a VPN connection because resolvd automates resolve.conf into a generated file from several other files and will overwrite any other functionality trying to add resolvers to resolve.conf, like an internal VPN company resolver for example.
Why is systemd even touching DNS?
Seems a little outside of it's scope.
This is kinda crazy to me personally when I found out resolvd was doing this.
Ideally, with systemd-resolved, you would have only 127.0.0.53 in your resolv.conf. (Anything that reads it directly is broken anyway).
Then you configure your DNS using NetworkManager, on respective connections. I.e. on your VPN, you configure the DNS on the other side of the connection, and (optionally) which zones it is supposed to resolve. (NetworkManager then configures systemd-resolved accordingly). The usage of this DNS then goes up and down together with your VPN connection. That's it.
I believe systemd-resolved is easily the absolute worst piece of systemd infrastructure. And especially so in Ubuntu! In 24.04 you can get away by uninstalling it then creating a sane /etc/resolv.conf; previous Ubuntu distros required a bit more than that.
Binding a DNS server to 127.0.01:53 is a perfect example of what has always bothered you about systemd? Seriously not getting it.
Providing DNS services on 127.0.0.1:53 has nothing to do with systemd. It's a convention used by NetworkManager, and associated network services. The very good reason being that network interfaces that are being brought up and down can choose to forward their DNS requests to 127.0.0.1, and allow a completely separate unit (resolved?) to deal with the details of where DNS requests get forwarded from there. (Or not, if they so choose).
Was there some OTHER burning compelling use-case that makes you want to use 127.0.0.1:53 for some other purpose?
> Was there some OTHER burning compelling use-case that makes you want to use 127.0.0.1:53 for some other purpose?
The problem is... by default... resolvd routes ALL DNS REQUESTS to this stub by over writing resolv.conf with ONLY that socket and forcing you to configure resolvers elsewhere. And you have to manually reconfigure this if you want to change it and it forces you to get in the weeds way to much when all you want to do is add a DNS resolver.
