← @LucasWerkmeistr Twitter archive

Lucas Werkmeister

@LucasWerkmeistr

Alright everyone, how about some systemd Tip Of The Day goodness?

I’ll start with some meta basics:

1/25/2017, 9:45:17 PM

Favs: 3

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

systemd.index(7): list of all systemd manpages (including documentation for C APIs). Explore! #systemdTOTD

1/25/2017, 9:47:47 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(Reminder: systemd.index(7) refers to manpage systemd.index in section 7, can be viewed with `man 7 systemd.index`. Section may be omitted.)

1/25/2017, 9:48:10 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd.directives(7): list of all systemd config file options, CLI options, journal fields, files, kernel cmdline options, … #systemdTOTD

1/26/2017, 10:30:08 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

What does MemoryHigh=1G mean? What does --firmware-setup do? What’s /etc/machine-info? systemd.directives(7) tells you where to look.

1/26/2017, 10:32:41 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemctl enable --now UNIT: enable and start unit in one command. Also works with disable (stop). #systemdTOTD

1/29/2017, 1:28:35 PM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

systemctl list-timers: list all systemd timers currently active, with times of last and next activation. #systemdTOTD

1/29/2017, 1:30:24 PM

Favs: 1

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(Add --all to list timers not currently running, too.)

1/29/2017, 1:30:37 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd manages coredumps for you: `coredumpctl list` lists coredumps, `coredumpctl gdb` debugs the last crashed process. #systemdTOTD

1/29/2017, 1:32:54 PM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

(Your binaries are probably stripped, but it’s still better than nothing, and you don’t have to go searching for the coredump file first.)

1/29/2017, 1:38:05 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd.exec(5) documents many sandboxing options that you can apply to services. I’ll focus on them for this week of #systemdTOTD

1/30/2017, 1:09:52 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

ProtectSystem+ProtectHome allow you to make /usr, /boot, /etc, /home+/root read-only or completely inaccessible for a service. #systemdTOTD

1/31/2017, 3:03:51 PM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

(You should put these in almost all your service files – services should never write to /usr, /boot or /home unless they do system updates.)

1/31/2017, 4:03:48 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

GENERAL DISCLAIMER: Always consult the manpage, I oversimplify for Twitter. For example, ProtectSystem+ProtectHome aren’t just booleans.

1/31/2017, 5:20:07 PM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

PrivateNetwork=yes: put a service into its own network namespace with no devices except loopback. No phoning home ;) #systemdTOTD

2/1/2017, 10:43:01 PM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

similarly, PrivateDevices=yes: give a service its own fake /dev file system with null, zero, random etc., but no real devices. #systemdTOTD

2/2/2017, 12:16:41 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can see the contents for yourself with:

sudo systemd-run -tq -p PrivateDevices=yes /bin/ls -l /dev

2/2/2017, 12:17:02 PM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

NoNewPrivileges=yes: ensure that a service (incl children) never gains new privileges (ignore setuid bit, file capabilities, …) #systemdTOTD

2/3/2017, 10:22:00 AM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

(The kernel feature behind this is

prctl(PR_SET_NO_NEW_PRIVS, 1);

see prctl(2) and Documentation/prctl/no_new_privs.txt.)

2/3/2017, 10:22:50 AM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

CapabilityBoundingSet: restrict the capabilities(7) of a root service (e. g. only CAP_SETUID, CAP_SETGID, CAP_SYS_TIME) #systemdTOTD

2/4/2017, 10:09:37 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For an example, see systemd-timesyncd:

systemctl cat systemd-timesyncd

2/4/2017, 10:10:12 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

AmbientCapabilities: grant selective capabilities(7) to a non-root service (e. g. CAP_NET_BIND_SERVICE, bind ports < 1024) #systemdTOTD

2/5/2017, 9:36:17 AM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

And that concludes systemd.exec(5) week! There’s a lot more in that manpage, and I might come back to it later, but let’s move on for now.

2/5/2017, 11:54:27 PM

Favs: 0

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

systemd-run(1): run a command like a service. Set properties with -p. Connect to terminal with -t. Tweak environment with -E. #systemdTOTD

2/6/2017, 12:18:28 PM

Favs: 0

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

For example, to see the effect of PrivateDevices:

sudo systemd-run -t -p PrivateDevices=yes ls /dev

(Add -q to suppress extra info.)

2/6/2017, 12:20:15 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Or to see the default environment of a service:

sudo systemd-run -t env

2/6/2017, 12:22:41 PM

Favs: 0

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

(That’s not *entirely* accurate, since -t adds a TERM. Without it, the output ends up in the journal, and you get a JOURNAL_STREAM instead.)

2/6/2017, 12:23:30 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can also use systemd-run to run a service later, e. g. --on-active=10m (in ten minutes) or --on-calendar=11:00 (11AM today) #systemdTOTD

2/7/2017, 10:52:03 AM

Favs: 1

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

In a way:
at = systemd-run --on-calendar
atq = systemctl list-timers
atrm = systemctl stop

2/7/2017, 10:54:23 AM

Favs: 0

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

Unit names can be abbreviated. sshd means sshd.service, /var means var.mount, /dev/sda means dev-sda.device. #systemdTOTD

2/8/2017, 10:16:28 AM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

You can use systemd-escape(1) to escape or unescape unit names. #systemdTOTD

$ systemd-escape -p --suffix=device /dev/sda
dev-sda.device

2/9/2017, 3:13:58 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Beware: systemd unit and configuration files do not support end-of-line comments! #systemdTOTD

# comment
Foo=bar # not a comment!

2/10/2017, 10:42:06 AM

Favs: 2

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

`systemctl status`, to show the status of a unit (its processes, journal entries, …), also accepts PIDs. #systemdTOTD

2/11/2017, 5:35:54 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For example, `systemctl --user status $$` shows me that my current shell is part of gnome-terminal-server.service.

2/11/2017, 5:36:41 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-inhibit(1) runs a command while delaying or blocking certain actions (sleep, shutdown, …). List locks with --list. #systemdTOTD

2/12/2017, 4:08:44 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For example, a quick-and-dirty way to temporarily disable sleep on lid close:

systemd-inhibit --what handle-lid-switch sleep infinity

2/12/2017, 4:10:16 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

GNOME uses this to take responsibility for keypresses away from logind :)

2/12/2017, 4:12:32 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

I quite liked the concept of a “topic week” (systemd.exec(5) last time), so let’s do that again! This week, journalctl(1).

2/13/2017, 6:44:32 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

journalctl -u UNIT: show all log entries for the specified UNIT. #systemdTOTD

journalctl -u sshd
journalctl -u logrotate.timer

2/13/2017, 6:46:00 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

journalctl -f: show recent journal entries and then keep printing new ones, like tail -f. #systemdTOTD

2/14/2017, 5:35:50 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

journalctl -b: select boot. -b -3 ≙ third boot before last, -b 5 ≙ fifth boot in journal. -b without arg means current boot. #systemdTOTD

2/15/2017, 2:38:17 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

journalctl --list-boots lists all boots in the journal with ID and offset.

2/15/2017, 2:38:39 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

If your --list-boots only shows current boot, try creating /var/log/journal/ – see Storage=auto in journald.conf(5) for why. #systemdTOTD

2/16/2017, 2:49:57 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

journalctl --since / --until: restrict to entries in a certain time range. Accepts strings like "3 days ago" or "-10m". #systemdTOTD

2/17/2017, 5:05:43 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For the full format of time specifications, see systemd.time(7), specifically the PARSING TIMESTAMPS section.

2/17/2017, 5:06:45 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

journalctl -e: show just the last few messages and scroll to end of pager. Control amount with -n (default: 1000). #systemdTOTD

2/18/2017, 10:29:41 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

The log output at the end of `systemctl status UNIT` is roughly equivalent to `journalctl -u UNIT -b -e -n 10`.

2/18/2017, 10:31:55 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

journalctl FIELD=VALUE: filter for messages with VALUE in FIELD. Processes can freely attach fields and values to messages. #systemdTOTD

2/19/2017, 3:14:36 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For example, in the upcoming Apache 2.6 you’ll be able to do this:

journalctl REQUEST_HOSTNAME=http://example.com REQUEST_URI=/foo/bar

2/19/2017, 3:16:03 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can list all available fields with journalctl -N, and all values for a field with journalctl -F FIELD.

2/19/2017, 3:16:50 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

journalctl’s awesome tab completion uses that to offer:

journalctl FOO<tab> # complete names
journalctl FOOBAR=<tab> # complete values

2/19/2017, 3:17:51 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Even programs that don’t use the journal API leave useful information in the fields:

journalctl SYSLOG_IDENTIFIER=thunderbird.desktop

2/19/2017, 3:20:33 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Attaching the same FIELD=VALUE to thousands of messages is dirt cheap, by the way, so no need to hold back there :)

2/19/2017, 3:33:28 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

The journal itself, for example, attaches multiple mostly-constant metadata fields to each message – see systemd.journal-fields(7).

2/19/2017, 3:34:36 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

How is it possible that fields are so cheap and can be searched for and enumerated nigh-instantaneously? You have one guess ;)

2/19/2017, 3:39:24 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-cat: run a command with output connected to the journal. #systemdTOTD

systemd-cat echo hi
echo hi | systemd-cat

2/20/2017, 5:59:54 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(There’s no way to tell from the process list because systemd-cat is just a bit of setup and then exec(), nothing sticks around.)

2/20/2017, 6:04:01 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can create a journal entry with arbitrary fields from a script using `logger --journald[=file]` (reads from file or stdin). #systemdTOTD

2/21/2017, 4:56:23 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Unit files can contain documentation links: #systemdTOTD

[Unit]
Documentation=https://example.com
Documentation=man:example(7)

2/22/2017, 3:48:24 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

`systemctl help UNIT|PID`: show the first manpage in Documentation= for UNIT (or PID’s unit). #systemdTOTD

systemctl help upower # e.g.

2/23/2017, 6:32:56 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

CORRECTION: All man pages of the unit are shown (concatenated), not just the first one. I was confused by the concatenation. Sorry.

2/23/2017, 7:16:01 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemctl mask UNIT: completely prevent a unit from being started (stronger than disable). With --now, also stop. #systemdTOTD

2/24/2017, 5:49:51 PM

Favs: 1

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

Like enable and disable, this is just a symlink under the hood – in this case, to /dev/null (from /etc or, with --runtime, /run).

2/24/2017, 5:50:41 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-path NAME: print system and user paths, taking into account env vars and some other conditions. Useful for scripts. #systemdTOTD

2/25/2017, 1:56:19 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Without argument, prints all paths.
--suffix can also be useful:

systemd-path --suffix myprogram user-binaries # install myprogram here

2/25/2017, 1:57:07 PM

Favs: 1

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemctl kill UNIT: Send a signal (--signal SIGNAL, default SIGTERM) to all (or --kill-who=main|control) processes of a unit. #systemdTOTD

2/26/2017, 6:25:59 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-cgls: show the cgroup tree and the processes of each cgroup. Can be more informative than parent process relationship. #systemdTOTD

2/27/2017, 8:15:17 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-cgtop: show resource usage of cgroups (not terribly useful unless you have accounting enabled for them). #systemdTOTD

2/28/2017, 10:38:44 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

By default, units only have task accounting. Enable more with e. g. CPUAccounting=yes for one unit or DefaultCPUAccounting=yes globally.

2/28/2017, 10:40:35 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Default*Accounting= is configured in /etc/systemd/system.conf; see systemd.resource-control(7) for the different controllers.

2/28/2017, 10:41:15 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

There are several signals you can send to PID1 if it doesn’t respond via systemctl (e. g. if dbus is hosed) – see systemd(1). #systemdTOTD

3/1/2017, 5:59:29 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

SIGTERM: daemon-reexec. Serialize state to file system, reexecute self, deserialize state again. (systemctl daemon-reexec)

3/1/2017, 6:00:18 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

SIGUSR1: reconnect to dbus.
SIGUSR2: dump state in textual form (presumably to kernel console).
SIGHUB: daemon-reload.

3/1/2017, 6:02:07 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

SIGRTMIN+20/21: enable/disable status messages on console
SIGRTMIN+22/23: set log level to debug/info

3/1/2017, 6:03:21 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

There are loads of signals for starting targets like halt/reboot/rescue, or for immediately halting/rebooting/etc. See manpage for details.

3/1/2017, 6:04:13 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(SIGHUB three tweets up should be SIGHUP, of course, sorry.)

3/1/2017, 6:05:19 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemctl list-dependencies UNIT: list dependencies of UNIT (duh). Found via @gdamjan on freenode – thanks! #systemdTOTD

3/2/2017, 5:59:49 PM

Favs: 1

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

With --reverse, show reverse dependencies (e. g. of /tmp, aka tmp.mount).
With --after/--before, show ordering dependencies.

3/2/2017, 6:01:12 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Journal messages can often be identified by the MESSAGE_ID field, a 128-bit ID. Much better than grepping the message! #systemdTOTD

3/3/2017, 11:25:10 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

As a user / administrator / whatever, you can see the MESSAGE_ID in output formats like -o verbose or -o json-pretty…

3/3/2017, 11:26:34 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

…and add a match for it to see only those messages:

journalctl MESSAGE_ID=7d4958e842da4a758f6c1cdc7b36dcc5 # systemd’s “Starting UNIT…”

3/3/2017, 11:27:02 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

And as a developer, you can generate new IDs for your messages with `journalctl --new-id128`, which prints a new ID in several formats.

3/3/2017, 11:27:43 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

`busctl tree` and `busctl introspect` let you explore dbus from the terminal. Try it out – busctl has AMAZING tab completion. #systemdTOTD

3/5/2017, 7:10:36 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

I’m serious: enter `busctl introspect`, hit tab a few times, and start exploring. Super easy. (Add --user for the user bus.)

3/5/2017, 7:12:50 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

With `busctl call`, `busctl get-property` and `busctl set-property`, you can access dbus from shell scripts. #systemdTOTD

3/5/2017, 7:15:58 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Example: #systemdTOTD

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager GetUnitByPID u $PID

3/6/2017, 5:54:11 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-analyze verify FILES…: verify unit files before loading them into the service manager. #systemdTOTD

3/7/2017, 10:08:21 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-analyze plot > plot.svg: emit an SVG image that shows startup times of units, highlighting units that delayed others #systemdTOTD

3/9/2017, 12:46:24 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Failed at step NAMESPACE spawning /usr/bin/…: No such file or directory

Possible cause: ReadOnlyPaths=/does-not-exist in unit #systemdTOTD

3/9/2017, 11:22:59 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

“No such file or directory” is obvious enough, but bridging from NAMESPACE to ReadOnlyPaths et al. can be tricky the first time you see this

3/9/2017, 11:23:49 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

All systemd tools read the environment variable SYSTEMD_LOG_LEVEL, e. g. `SYSTEMD_LOG_LEVEL=debug journalctl`. #systemdTOTD

3/10/2017, 10:22:26 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(Another tip from #systemd on Freenode – thanks Mic92!)

3/10/2017, 10:23:10 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

The available values of SYSTEMD_LOG_LEVEL are the classic syslog(3) log levels: debug, info, notice, warning, err, crit, alert, emerg.

3/10/2017, 10:24:50 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can also add SYSTEMD_LOG_LOCATION={yes,no} to include source code location of message, if you want to investigate further. #systemdTOTD

3/12/2017, 1:56:47 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Also, SYSTEMD_LOG_TARGET={console,kmsg,journal,syslog,auto,safe,…}. auto = console/journal/kmsg. safe = console/kmsg. #systemdTOTD

3/12/2017, 1:00:14 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

And finally, all systemd services also parse the kernel commandline for these options: systemd.log_{level,location,target}. #systemdTOTD

3/13/2017, 4:46:57 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemctl show-environment: show service manager environment. Particularly interesting with --user in a graphical session. #systemdTOTD

3/14/2017, 6:21:15 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd and its daemons load configuration from /usr/lib, /run, /etc, in that order. (Distros without /usr merge: just /lib.) #systemdTOTD

3/15/2017, 10:48:40 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Distribution packages install into /usr/lib. Dynamic generators write to /run. System administrator overrides both in /etc.

3/15/2017, 10:49:11 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You should never, EVER have to edit a file in /usr/lib. Copy it to /etc and edit it there. No more conflicts on package upgrade.

3/15/2017, 10:50:29 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Or, better yet, keep the OS-provided file and augment it with overrides in /etc/…/$filename.d/$whatever.conf.

3/15/2017, 10:51:39 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

`systemctl edit UNIT` can be used to create such override files, or (with --full) copy the original file and edit it. #systemdTOTD

3/16/2017, 11:25:04 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

User manager follows similar logic, so $XDG_RUNTIME_DIR/systemd/user is a convenient place to play with unit files. #systemdTOTD

3/18/2017, 1:16:44 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You don’t need root since it’s just your own instance, and it’s in /run so files don’t stay around forever if you forget to delete them.

3/18/2017, 1:17:15 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

I want to talk about overrides a bit more, since they’re pretty useful.

3/19/2017, 12:35:57 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Any *.conf file in a FOO.d/ directory in the systemd load path will augment the original FOO file. #systemdTOTD

3/19/2017, 12:36:50 PM

Favs: 1

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For example, you can add sandboxing to sshd.service in /etc/systemd/system/sshd.service.d/sandboxing.conf.

3/19/2017, 12:37:58 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(Don’t forget the .service suffix! /etc/systemd/sshd.d/sandboxing.conf won’t work.)

3/19/2017, 12:38:22 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Overrides of the same name in different load paths override each other, just like regular files.

3/19/2017, 12:39:28 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For instance, symlink /etc/systemd/user/dbus.service.d/flatpak.conf to /dev/null to kill /usr/lib/systemd/user/dbus.service.d/flatpak.conf.

3/19/2017, 12:40:08 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

`systemctl set-property UNIT Foo=x Bar=y…` can be used to set properties on services. They are stored in overrides in /etc. #systemdTOTD

3/19/2017, 12:41:21 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(Or, with --runtime, in /run, so they’re lost on reboot – temporary.)

3/19/2017, 12:41:47 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-delta: show overriden and drop-in configuration files. (Without any arguments: show all of them.) #systemdTOTD

3/20/2017, 9:32:32 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-delta /etc: only changes in /etc (system administrator)
systemd-delta systemd/user: changes to user units (no matter where)
# …

3/20/2017, 9:33:54 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd exposes many cgroup resource control settings (e. g. CPUQuota=50%, MemoryMax=1G). See systemd.resource-control(5). #systemdTOTD

3/21/2017, 11:59:41 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can also apply most of them to a running service, e. g. `systemctl set-property -p TasksMax=100 apache2`. #systemdTOTD

3/22/2017, 12:05:26 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(For most other properties, the change is only applied the next time they’re started.)

3/22/2017, 12:05:48 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd.resource-control(5) isn’t just for services: you can also apply those settings to slices, which group multiple units. #systemdTOTD

3/23/2017, 9:27:53 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

By default, system services go in system.slice, user sessions in user.slice, and containers in machine.slice…

3/23/2017, 9:28:56 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

…but you can also configure your own slice units (foo.slice in a unit load path) and add units to it with Slice=foo.slice.

3/23/2017, 9:29:31 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Also, instanced services get a per-template slice by default (e. g. system-tor.slice for tor@.service, system-getty.slice for getty@, …).

3/23/2017, 9:36:18 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(CORRECTION: an earlier version of that tweet read “instanced units”, but according to manpage that’s only for services, not other units.)

3/23/2017, 9:37:03 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can configure sysctl parameters in /etc/sysctl.d/*.conf files – systemd-sysctl.service will read and apply them on boot. #systemdTOTD

3/24/2017, 11:42:55 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

This follows the same /etc > /run > /usr/lib logic as everything else, so packages can place their files in /usr/lib/sysctl.d/60-mypkg.conf.

3/24/2017, 11:43:42 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Which is also its big advantage over plain old /etc/sysctl.conf – that’s a single file, packages can’t edit it without risk of conflicts.

3/24/2017, 11:44:22 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

If a package installs /usr/lib/sysctl.d/60-foo.conf, you can override it very simply by symlinking /etc/sysctl.d/60-foo.conf to /dev/null.

3/24/2017, 11:44:58 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Or place a different file there, with amended content. Files of the same basename override each other.

3/24/2017, 11:45:37 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

I forgot to update this yesterday and it’s almost midnight, so let’s have several updates in a row about systemd’s container support.

3/26/2017, 11:43:37 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-nspawn(1) can be used to run a system image on disk as a container. Specify root dir with -D or image file with -i. #systemdTOTD

3/26/2017, 11:43:47 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

By default, it will just set up the namespaces and then drop you into a shell (or run a specified command); add -b to boot a full init.

3/26/2017, 11:44:25 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Alternatively to -D or -i, you can specify a machine name with -M. Machines are stored under /var/lib/machines and managed by machinectl(1).

3/26/2017, 11:44:36 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

`systemctl start|stop MACHINE` manages such a container as a system service (internally, systemd-nspawn@MACHINE.service). #systemdTOTD

3/26/2017, 11:44:50 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

`systemctl clone NAME1 NAME2` clones an image. If /var/lib/machines is btrfs, creates a subvolume (very efficient). #systemdTOTD

3/27/2017, 12:01:26 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(In fact, if /var/lib/machines is blank and non-btrfs, machinectl will mkfs.btrfs /var/lib/machines.raw and loop mount it there.)

3/27/2017, 12:01:33 AM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Use `systemd-nspawn --ephemeral` to run a temporary copy of an image. Again, super efficient if the backing store is btrfs. #systemdTOTD

3/28/2017, 8:26:56 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For example, I have a Debian-Jessie-Base image that I use like this:

sudo systemd-nspawn -M Debian-Jessie-Base --ephemeral /bin/bash <build

3/28/2017, 8:28:32 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can also mark an image as read-only with `machinectl read-only IMAGE true|false`. #systemdTOTD

3/29/2017, 9:00:30 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

For example, my aforementioned Debian-Jessie-Base image is read-only, so I don’t accidentally pollute it when I forget a --ephemeral switch.

3/29/2017, 9:01:01 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Every now and then, I’ll make it writable again (machinectl read-only DJB false) and update it (systemd-nspawn -M DJB apt update,upgrade).

3/29/2017, 9:02:29 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

machinectl can also manage images with {pull,import,export}-{tar,raw}. #systemdTOTD

3/30/2017, 11:56:51 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(I must admit I’ve never used this feature, because I couldn’t find suitable URLs to pull – most distros publish installers, not images.)

3/30/2017, 11:57:55 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

machinectl set-limit [NAME] BYTES: limit the size of a single container image, or the total size of all of them. #systemdTOTD

3/31/2017, 11:52:24 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(Running this command on an empty non-btrfs /var/lib/machines will, as mentioned before, create and loop-mount a btrfs filesystem for it.)

3/31/2017, 11:53:07 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

`machinectl shell NAME` gets you interactive root shell on a running container. Specify another user with USER@NAME (like ssh). #systemdTOTD

4/1/2017, 11:40:08 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

If you omit the machine name (i. e. `machinectl shell user@`), it defaults to the host system (called .host).

4/1/2017, 11:45:40 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Somehow, this turned into “systemd wants to replace su”, which is such an astonishingly bad take I really don’t understand how it happened.

4/1/2017, 11:47:05 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

But that’s still the second Google result for “machinectl shell”. It’s ridiculous. No, nobody wants to replace plain old `su`.

4/1/2017, 11:48:05 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(The same thing happened to systemd-mount as well, but we’ll get to that later. Spoiler: no, `mount` isn’t being replaced either.)

4/1/2017, 11:49:22 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

You can also run a command in a container with `systemd-run -M NAME COMMAND…`. (systemctl, journalctl etc. also support -M.) #systemdTOTD

4/2/2017, 11:09:04 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

So what’s the difference between `machinectl shell NAME COMMAND…` and `systemd-run -M NAME COMMAND…`?

4/2/2017, 11:09:42 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

machinectl shell is intended more for interactive usage (COMMAND can specify an alternative shell), systemd-run for scripts.

4/2/2017, 11:10:10 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

systemd-run can propagate the exit status with --wait (which machinectl shell doesn’t). It also offers more configuration options.

4/2/2017, 11:11:26 PM

Favs: 0

Retweets: 0