I’ve spent quite some time on trying out systemd recently. Here are some thoughts.
Compared to ordinary SysV init systems, you will never really know when systemd will start your activated service units. This is part of its nature, though, and simply by design. If you want your init system to launch your different jobs in parallel to each other, this issue will be inevitable. This was never the case with SysV as everything was launched in a sequential order; awesome-daemon placed as 78-awesomed would be started after 77-foobard and just before 79-otherd.
Of course, if you’d have a really strict systemd unit setup, you could probably make systemd behave like this as well, simply by ordering it manually by adding a single “Before” and “After” option into each unit file. Then things would probably launch sequentially. Well, there are couple of other things to keep in mind also, of course, since systemd can be configured in zillions of ways. But, really? Why would you even use systemd if you want to keep a non-parallized boot sequence?
Using the “Before”, “After” and “Requires” options in your unit files is how you manage your boot order. Doing this is necessary since some services will always depend on other units to have been completed, etc. For example, you’d probably don’t want to launch your http daemon if your web root filesystem isn’t mounted yet.
But what if you add these options, and whatever you’re trying to launch seems to have failed to start? Have a look at the systemd journal using the journalctl command.
Will it show anything useful? It won’t show you what the boot order actually looks like, that’s for sure. The journal shows things like:
- Activated units failing (e.g. missing binary or not executable):
systemd[419]: Failed at step EXEC spawning /bin/test: No such file or directory - Any output of launched (non-forked) programs
- As explained below, any syslog messages
But it won’t show you verbose messages like “starting service x now”. It seems like the only actual way to show what the boot order looks like is to generate a graph of this in SVG format. Just run “systemd-analyze plot > plot.svg”. Oh wow, splendid, I’ll definitely do that the next time I’m in a data center, debugging some server system… >_>
It’s pretty interesting though, here’s what it looks like:
And then there’s this dependency map you can create using the dot utility:
systemctl dot | dot -Tsvg > systemd.svg
This map is immense:
OK. That’s cool and all, but what about some simple logging as well – what if the journal also showed stuff like this?
11:31:41 awesomed: Starting
11:31:41 crapd: Failed (ExecStart=/crap/path: file not found)
11:31:41 foobard: Starting
11:31:41 otherd: Starting
11:31:43 awesomed: Successful (forked)
11:31:43 depd: Starting
11:31:43 depd: Successful (forked)
11:31:45 foobard: Successful (exited: return code = 0)
11:31:46 otherd: Failed (exited: return code = 1)
Lets say “depd” was set up to start after “awesomed”. This way you’d easily know if it actually turned out that way. Is it really that difficult to implement? I gotta have a look at that code some day.
Fine, I’m not aware of any server systems that run systemd yet, so I probably don’t have to worry about that situation anyway.
systemd comes with some very nice features that are especially useful in server systems though, such as built-in process monitoring. No longer would you need to run software like monit, for example. Monit would check every x seconds if a process is still alive, unlike systemd that would instantly detect the terminated process – simply as it continuously monitors all of its “children”. It’s probably both faster and more reliable.
In server systems you’d often like to restrict resource usage – systemd makes this easy; CPU, I/O and memory restrictions are easily set directly in the service unit file. You can even set the OOM badness score of a daemon this way. No more database servers getting unexpectedly terminated by an untamed OOM killer…
Please I have children! Ok, no, but I might have some volatile data though, so please terminate my http daemon instead of mysqld… =)
Of course, you could do these things with ordinary initscripts as well, but have you ever bothered to implement it? systemd makes these things, and so much more, so very easy to handle. It’s pretty cool. Oh snap, I’m starting to think systemd is a good thing, after all.
Then we have all these other system daemons that systemd provides replacements for, like crond, atd, inetd and syslog. Since systemd obviously knows how to execute, manage and monitor processes, running scheduled jobs and one-shot network processes like this is probably only a good thing. But even the syslog part makes sense.
By implementing syslog directly into systemd, all data that should be logged will simply be logged. With SysV, the syslog daemon is started among the other system daemons, and who knows what sort of data we miss out on pre-initialization of the syslog daemon? If you’d like to run an ordinary syslog daemon, you can do this as well, it just needs to listen to systemd’s syslog socket instead of the ordinary one.
Actually though, I’m not really sure why you’d ever need any other syslog daemon. The systemd one will even forward your log messages to your syslog server if you’d like, which is probably the only thing you need on a thin server.
But back to business… I run Qubes with Fedora VM templates and this is mainly where I get to use systemd (oh right, I got some Arch Linux installation running systemd as well though). Debugging the boot process is really not that fun, even though I have access to a SVG viewer in these cases (>_>).
I’ve updated my previous article systemd rc.local service with Qubes with quite some additional details.
Especially with the update of the rc.local article, I have a single recommendation when it comes to setting up systemd services:
Really learn the difference between the services types (“Type” option) – see man 5 systemd.service. By doing this you won’t get into all sorts of unbelievable foobar problems. And you won’t start thinking “I’m gonna create my own fscking init system”. =)
Frankly, I must say that systemd seems alright and I think it’s here to stay. Its command line tools are sometimes a bit horrid though, but that can always be fixed.