Why I do not like systemd
I have overall unsupported setup in Fedora. I do have RAID and I am booting of it, I have i686 architecture and I want to use syslog and I do hibernate and want instant networking. This all is now unsupported on Fedora. How I found it?
RAID
Bug: "Can not boot from mdraid with LVM on top of it"
Solution: WONTFIX - Yes, it needs manual intervention. Sorry for the inconvenience.
What is the problem? Dracut - from some time - does not include automatically the "rd.auto" or similar in initramfs, now it is up to anaconda to include that at installation time. I.e. any boottime auto setup wont work.
I put the rd.auto=1 into /etc/default/grub but newely installed kernels still do ignore it...
You may ask what systemd has to do with that? Almost nothing, except it is not able to boot the machine and it gets stuck. STUCK. The only good thing is my bug report:
Bug: "systemd can not keyboard reboot if ctr-alt-delete.target is not present"
Solved as: "RFE: Pressing C-A-D three times within 1s should dump systemd's state to console, pressing it 10 times within 1s should reboot the machine hard" after more than 1.5 year, so I can now put all my anger into hitting the C-A-D. I found this solution good for that reason now, while I considered it bad idea before.
Syslog
...at times of all mighty journald too old to be used.
My Question: "How to correctly disable journald"
Was answered: This is simply not supported.
and it is true. All my attempts to disable journal were unsuccessful. Most close is to set the
Storage=none
however this leads to logs being everywhere but most often nowhere. With each release of Fedora I have to fight again to have a log in text files instead journal (you know - ForwardToSyslog = yes was changed to "no" by default etc.).
Now I needed the logs from some boot failures. I opened up the /var/log/message and the only thing I got is
rsyslogd-2177: imjournal: 178981 messages lost due to rate-limiting
journald flushed all the logs at once... great. Nothing logged. And this happens all the time.
All over the rescue shell and so on, I read hints written to console like "use journalctl -xe to get..." but I do get nothing from them as I do not have a journal enabled. Also there is a new "README" in /var/log:
You are looking for the traditional text log files in /var/log, and they are gone? Here's an explanation on what's going on: ...
...
Today I was investigating one of the computers crashing regularly. I found this in the log file:
... May 15 16:11:19 mach lircd-0.9.2a[1189]: Error: could not open /dev/lirc0 May 15 16:11:20 mach lircd-0.9.2a[1189]: default_init(): Permission denied May 15 16:11:21 mach lircd-0.9.2a[1189]: Error: could not open /dev/lirc0 May 15 16:13:18 mach journal: Permanent journal (/var/log/journal/) is currently using 2.4G.#012Maximum allowed usage is set to 2.4G.#012Leaving at least 3.6G free (of currently available 3.9G of space).#012Enfo rced usage limit is thus 2.4G. May 15 16:13:25 mach kernel: audit: type=1701 audit(1463321483.453:730): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=425 comm="systemd-journal" exe="/usr/lib/systemd/systemd-journald" sig=6 May 15 16:13:26 mach kernel: gnome-shell invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 May 15 16:13:26 mach kernel: gnome-shell cpuset=/ mems_allowed=0 May 15 16:13:26 mach kernel: CPU: 2 PID: 2204 Comm: gnome-shell Not tainted 4.4.6-201.fc22.x86_64 #1 May 15 16:13:27 mach kernel: Hardware name: /D54250WYK, BIOS WYLPT10H.86A.0021.2013.1017.1606 10/17/2013 May 15 16:13:27 mach kernel: 0000000000000286 0000000012f3b740 ffff880119a0ba38 ffffffff813b542e May 15 16:13:27 mach kernel: ffff880119a0bc30 ffff88009e93da00 ffff880119a0baa8 ffffffff8122acf3 May 15 16:13:28 mach kernel: 01ff880119a0ba58 0000000012f3b740 00000000ffffffff 0000000000000000 May 15 16:13:29 mach kernel: Call Trace: ... May 15 16:13:55 mach kernel: Out of memory: Kill process 10658 (inkscape) score 292 or sacrifice child May 15 16:13:55 mach kernel: Killed process 10658 (inkscape) total-vm:3000488kB, anon-rss:1737656kB, file-rss:0kB May 15 16:13:55 mach journal: Journal started ...
There was a flood of logs from licrd going crazy for some reason, but the result? When journal triggered some "cleanup" it started OOMkiller, that killed everyhing. This logging is broken by design!
While digging into this I run into another issue I hate about this:
messages-20141123.gz messages-20141130.gz messages-20160424 messages-20160502 messages-20160508 messages-20160515
Looks weired right? Yes, for some reason logs stopped to rotate and compress more than a year ago. On brief look, you just miss this, as the old compressed logs sit there... I have no idea why they stopped to compress all of a sudden.
Hibernate
Supported? Well read that
Bug: "After upgrade hibernate no longer works"
Why: There was na update to systemd that solves everybody's problem with fsck on resume. Now resume rather does not work at all. There is one more thing:
Bug: "RFE: the kernel used during hibernating should be marked and used when resuming"
Why: I never had any serious issues booting the different kernel on resume compared to that used on suspend until systemd came. If you do systemd update or you do "incompatible" kernel update the resume results in horrible things happening including file system corruptions. It gets slightly better now, the system just resets during boot, but it could be a coincidence not a fix.
Networking
Bug: "NetworkManager is stuck for a minute after resume before it finishes and fills the resolv.conf
Reason: There is a dispacher script for ages that does
case "$2" in
up|down|vpn-up|vpn-down)
/bin/systemctl try-restart sendmail.service || :
;;
esac
but now it waits a minute to resolve localhost that can not be resolved while networking is not up... I thougth this kind of issues were solved with systemd. Nope. They just became impossile to debug.
Failed service = you are helpless
Today I got another boot problem - system dropped me to a rescue shell due to failing rpc-statd.service (I do not know why yet). When I entered the rescue shell, my attempts to use systemctl ended in most cases with
Error getting authority: Error initializing authority: Could not connect: No such file or directory (g-io-error-quark, 1)Reason: system was not fully booted, not all services are running and think what - you can not use systemctl to manage services becase required services were not started.. so how do you start them? You can not.
iptables
With new xbtables locking feature, runnig more services in parallel, that set iptables rules, you may see some of them are randomly not applied. This is due to systemd parallel service startup if you do not use an option -w with your iptables commands.
Another interesting story - systemd depends on iptables, if you want to use nftables you are out of luck. You can not get rid of iptables.
dbus restart
Restarting a DBUS service leads to very weired system behavior. Without e.g. restarting systemd-logind there is very long delay at login.
loginctl Failed to list sessions: Activation of org.freedesktop.login1 timed out
logind terminates user processes on logout
Great idea to kill all user processes at logout. What about screen, tmux, nohup... need to be run with "systemd-run".
need Xorg log?
With every OS version there is a different magic:
journalctl -b _COMM=gdm-x-session (Fedora 22+) journalctl -b _COMM=Xorg.bin (Fedora 21) journalctl -b _COMM=Xorg (Fedora 20 and earlier)
Simple right? Not enought? Well you may find the Xorg logs on the disk - for gdm session it is somewhere under /var/lib/gdm/.local/xorg/Xorg.0.log but no timestamps, users have their own session logs under /home/user/.share/xorg/... and hey, maybe this log is from current session. Just maybe. Well you see there -- Reboot -- in the log? This is not exactly reboot well, just another Xsession start when user logs in. Sorry for confusion. This is the way we wanted to solve the hard to understand world of linux distributions. While in past everybody looked in /var/log/Xorg.0.log now just everybody do not know - so they can no complain there is some error in the log.
Killing thousand of processed = systemdeath
On one system there was a periodical read of few bytes from USB. Due to new version of libusb this did not work. The process did not finished and there started to grow number of process up to several thousands. As the memory consumption was minimal, the system worked normally until and attempt to kill those processes. The system got almost unresponsive. Rebooting it from console took more than 20minutes, was not successful after all and some weired things happened to systemd:
systemd-journald[13542]: File /var/log/journal/04aa..../system.journal corrupted or uncleanly shut down, renaming and replacing. systemd-coredump[22374]: Detected coredump of the journal daemon itself, diverted to /var/lib/systemd/coredump/core.systemd-journal.0....xz. ... systemd-logind[12074]: Failed to abandon session scope: Transport endpoind is not connected
Another nice reboot thing - trying to reboot remotely system with load 842 (all attempts to kill the processes causing the load failed) - no go:
[root@server ~]# reboot Failed to start reboot.target: Connection timed out See system logs and 'systemctl status reboot.target' for details. [root@server ~]# systemctl status reboot Failed to get properties: Connection timed out
Broken by design^2.
A start job is running for dev-disk ...602
I made a copy of disk with dd, then removed swap and extended a partition before swap, over the removed swap partition on the new disk and booted up the system. Now the fun begins, during boot I saw twice the message systemd is looking for a dev-disk\horribleChars..602. After boot the swap _was_ attached to the system. I do not know how systemd found it, but it mounted non existing partition in the middle of existing partition. I made a swapoff, made a new partition for swap, changed the fstab - can not mount it, need to regenerate the systemd unit or what the heck, then rebooted. Now swap is on OK, but boot is still using the cryptic old UUID to look for a swap. After all I found, due to hiberantion issue (see above) I had to attach the resume=UUID to kernel command line. I tried to change this in the grub at boot, but you know - are you able remember the UUID? No? Bad luck. I had to change the /etc/default/grub, reinstall grub (as the one I used was too old) and regenerate the config. Now finally the systemd is not waiting 90s at boot for the non existing partition.
Related to this is a disk failure in the e.g. CEPH storage. If you remove and reattach the device, it is immediately unmounted by systemd even thou it has the same UUID, until you issue a daemon-reload and systemd regenerates the .mount files.
Consoles on demand...
There a great new improvement by systemd - it does not start TTY consoles right at the beginnig, but on demand when you switch to a different VT (except there are _again_ exceptions for tty1 and tty6...sigh). Authors say that "This behaviour is mostly transparent to the user: if the user activates a VT the getty is started right-away, so that the user will hardly notice". This a bullshit. I have servers running for days that when you come and switch console you stare on the blank screen for several seconds. Now lets have a machine that is overloaded.. when console will never start as it needs the resources that systemd saved... knock knock.
Well and how about a serial console? It is only started if you add a kernel parameter console=ttyS0 automaticaly. Otherwise you need to start it with systemctl start serial-getty@ttyS0.
Service file as a symlink?
This is how systemd chase its tail with symlinks: systemd lost in the symlinks Or maybe symlink or not symlink...
Systemd timers
For ages I used cron jobs to do the... jobs. Now I found for 3 month my mlocate databases were not update. What the hell? The mlocate package switched to systemd timers (mlocate-updatedb.timer) and tadaaa... the timer was not even enabled not even started.
systemctl enable mlocate-updatedb.timer systemctl list-timers --all NEXT LEFT LAST PASSED UNIT n/a n/a n/a n/a mlocate-updatedb.time systemctl start mlocate-updatedb.timer
What is worse, the timers do not send any mails in case of failure, they may spit something in the log. There is some crunky way to do this via OnFailure=.
2019: the systemd timers still is not able to replace anacron but as the jobs are not in the cron anymore, they are just not executed at all..
2021 Next chapter - the debian has by default apt-daily and apt-daily-upgrade timers.. one would think they do the job right - update the packages. NO. Those are timers, that calls a wrapper /usr/lib/apt/apt.systemd.daily around unattended-upgrades, but if this package is not installed, it just silently does nothing. As my experience with unattended-upgrades is rather bad, I switched back to cron-apt.
Login defs
systemd-sysusers do not respect the login.defs limit for UIDs creating overlapping user/group IDs.
No support for keyscript in crypttab
Had a partition the needs to run a script in crypttab to get the key? You are out of luck. Systemd supports only keyboard passphrase or static keys.
You had a separate /var? Yes you had.
Use e.g. encrypted NFS, you may be out of luck soon:
https://lists.freedesktop.org/archives/systemd-devel/2016-February/035812.html
Now how to solve a /var/ on a e.g. /home? Well it was easy, now you need a hack in fstab! From systemd 229
# /etc/fstab home/var /var x-systemd.requires=/home,x-systemd.automount,none bind 0 0Is this even possible?! This happens always when you think you will create a "system" that does everything possible in "LotOfWordKyesNecessary" language.
Journald sorts by time.. always
I had a system that exhibits a problems with RTC, but finding what came when to the log is impossible, as journal always sorts the messages by time, you can not display the log messages in the order they came. In usuall syslog scenario, I would see a messages with newer time being before those with older time, but this is not possible with journal.
i686
Well this is overall bad idea to use i686 with 1GB of RAM at times of x86-64... you know. It is too old, and only 15% of machines use it nowadays. Drop it. it has nothing to do with systemd - I can not believe that.
Can not umount chrooted environemnt
As soon as you do
mount -t proc /proc proc/ mount --rbind /sys sys/ mount --rbind /dev dev/
systemd "notices" the newly mounted directory for something inside and you won't be able to unmount all of them, because systemd makes them busy.