I’m running multiple Docker containers such as Pi-Hole on an Odroid XU4 with Arch Linux ARM.
After updating to Systemd version 248 last month, I noticed that all my Docker containers started throwing an error:
ERROR: for <container name> Cannot start service <container name>: failed to create shim: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: invalid argument: unknown
After some reading, I discovered that with Systemd 248 cgroup v2 became the default and all containers that do not yet support it, will fail to load.
I wasn’t the only one, as other people on the Arch Linux ARM forums also complained about a similar issue.
This article explains in great details all about it, but what I was actually interested in was getting my Docker containers back and running.
Both that article and the Systemd man page mention the kernel command line (also known as “kernel parameter”) systemd.unified_cgroup_hierarchy.
Quite naively, I thought to myself that running sysctl systemd.unified_cgroup_hierarchy=0 would let me disable it at runtime to test that it actually fixes the problem.
However, I quickly realized this wasn’t that easy:
$ sysctl systemd.unified_cgroup_hierarchy=0
sysctl: cannot stat /proc/sys/systemd/unified_cgroup_hierarchy: No such file or directory
Asking around on Reddit I learned that there are parameters that cannot be set at runtime, and this was one of them.
Oh well… I try as hard as I can to avoid messing with my cmdline (references here and here), but I went ahead and changed it.
Every bootloader has a different way of setting its cmdline.
The Odroid XU4 uses uboot, so I had to edit /boot/boot.txt:
setenv bootargs "... systemd.unified_cgroup_hierarchy=0"
Then, I had to run the mkscr script in /boot, only to find out that I’m missing the uboot-tools package:
$ ./mkscr
mkimage not found. Please install uboot-tools:
pacman -S uboot-tools
Installing it and rebooting, I could confirm that the parameter was correctly set:
$ cat /proc/cmdline
... systemd.unified_cgroup_hierarchy=0
I ran Docker and… Success!