Update overwrites your .service: systemd drop-in directories and three commands to verify

The Xray installer rewrites /etc/systemd/system/xray.service on every update, and everything you manually appended to this file disappears silently. The XTLS script warns about this directly in a comment inside its own drop-in file: «Or all changes you made will be lost!». The story with the package’s docker.service is the same — the update will revert to the vendor version.

The standard systemd response is not “copy the unit to /etc and edit the copy”: the copy gets detached from the upstream and quietly becomes obsolete after a couple of releases. The tool is called drop-in: a small .conf next to the unit, overriding individual lines.

How systemd builds the final unit

For foo.service, the manager looks for the directory foo.service.d/ and after parsing the main file, it merges in all files with the suffix .conf. There are three directories, with increasing priority: /usr/lib/systemd/system//run/systemd/system//etc/systemd/system/. In systemd.unit(5) this is stated as: “Drop-in files in /etc/ take precedence over those in /run/ which in turn take precedence over those in /usr/lib/”.

A detail that is often overlooked: the order of application is defined by file names, not by directories — “Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.” Therefore override.conf, which is created by systemctl edit, will come after the install-time 10-donot_touch_single_conf.conf: ASCII digits come before letters.


Diagram from systemd.unit(5) documentation

Info:

Besides named directories, there is a top-level service.d/ — immediate for all system services. For template instances, foo@bar.service.d/ is checked, then foo@.service.d/; units with dashes get shortened variants like foo-.service.d/.

Proxying the Docker daemon without editing docker.service

The daemon reads not ~/.docker/config.json, but the environment variables of its process, so the official documentation recommends drop-ins:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/http-proxy.conf >/dev/null <<'EOF'
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:3128"
Environment="HTTPS_PROXY=http://proxy.example.com:3128"
Environment="NO_PROXY=localhost,127.0.0.1,.corp"
EOF
sudo systemctl daemon-reload
sudo systemctl restart/docker

The control command — systemctl show --property=Environment docker: should return a line with three pairs. If empty, the file is not in the right directory or a daemon-reload wasn’t performed. In rootless mode, the directory is different: ~/.config/systemd/user/docker.service.d/, with commands using --user and without sudo.

Overriding the startup command: why ExecStart is cleared first

Here is the main trap. ExecStart= is a list option: a line from a drop-in does not replace the original, it is appended to it. And systemd.service(5) requires: “Unless Type= is oneshot, exactly one command must be given.” Two commands on a normal service will prevent the unit from loading.

First, the list is cleared with an empty assignment — “If the empty string is assigned to this option, the list of commands to start is reset” — and only then the own one is set:

[Service]
ExecStart=
ExecStart=/usr/local/bin/xray run -confdir /usr/local/etc/xray/

This is exactly how the stock Xray drop-in is structured: two lines, the first empty. The same rule applies to ExecStartPre=, Environment=, After= and other accumulating options.

Error:

Do not edit /usr/lib/systemd/system/*.service directly: the directory belongs to the package manager. And do not copy the unit wholesale via systemctl edit --full just for one line — the copy will stop receiving upstream fixes, and you’ll find out about it on the next broken release.

Limits and restart policy: you do not need to reset

These are standalone options: the last assigned value wins.

sudo systemctl edit --drop-in=90-limits.conf xray.service
[Service]
LimitNOFILE=1048576
Restart=always
RestartSec=5

systemctl edit reloads the configuration after leaving the editor — “equivalent to systemctl daemon-reload”. They check not the file, but what the manager has applied: systemctl show -p LimitNOFILE -p Restart xray.service. For comparison, the Xray installer already sets LimitNOFILE=1000000 and LimitNPROC=10000 — you don’t need to raise the descriptors, but Restart=on-failure to always is often changed.

Success:

Three commands after any change:

  • systemctl cat xray.service — shows the main file and all drop-ins with full paths, in the order of application;
  • systemd-analyze verify xray.service — will find syntax errors before restart;
  • systemd-delta — shows across the system what is overridden.
Warning:

systemctl revert xray.service reverts the unit to the vendor state — and deletes the directory xray.service.d/ entirely, along with other drop-ins. For Xray this means losing the installed 10-donot_touch_*.conf with the path to the config. If you only want to remove your own changes, delete your file and run daemon-reload.

A separate scenario is the --runtime flag: changes move to /run/systemd/system/ and will not survive a reboot. Handy when you need to keep the service with Restart=no for a while and ensure you don’t leave anything behind.

Related on the forum: how journald is organized and what to configure in it and accepting a new VPS in the first hour.

Sources

Question:

Do you keep your unit edits in separate drop-ins, or do you edit the copied file wholesale? And was there ever a case where a package update silently restored the vendor’s .service and something stopped working as a result?