My new monitoring paged me twice on day one. Good.

The hardening work added two new scheduled jobs: a 5-minute heartbeat to healthchecks.io and a nightly restic backup. Both had passed their tests. Within 24 hours, both triggered a Discord alert.

Alert 1: the heartbeat "failed"

archlinux: Failed systemd units: server-heartbeat.service

The server was fine. The journal showed two failed pings to hc-ping.com: a timeout, and then an HTTP/2 error on the next try. My own connection looked healthy (0% loss to 1.1.1.1 at about 2.5 ms), but the route to the ping service was about 240 ms away and dropping a lot of ICMP.

The real bug was the design. The heartbeat script exited non-zero when a ping failed, so systemd marked the unit failed, and my existing health monitor (which reports failed units to Discord) raised the alarm.

But a missed ping is already the signal. healthchecks.io alerts when pings stop arriving, so making the unit fail too just creates a second, noisier alert. The fix:

• ping over HTTP/1.1, which avoids the HTTP/2 failure;
• retry up to 5 times, 3 seconds apart;
• never fail the unit because a ping failed. If the pings really stop, healthchecks.io alerts after the 10-minute grace period.

The backup job's pings got the same treatment.

The good news: the alert chain worked end to end. A problem was detected and reached my phone within minutes.

Alert 2: the backup failed at 03:36

⚠️ archlinux: Failed systemd units: server-backup.service

This one was a genuine failure, and nothing had been backed up that night. restic stopped within a second because it couldn't find a cache directory. When systemd runs a service, HOME isn't set, and restic puts its cache under $HOME/.cache.

The first run had worked because it was started by hand from a shell that has a home directory. The fix was one line in the unit and one in the script:

# /etc/systemd/system/server-backup.service
[Service]
Environment=HOME=/root

# /usr/local/bin/server-backup
export RESTIC_CACHE_DIR=/var/cache/restic

Running it exactly the way the timer does took about 12 seconds, the repository check came back clean, and the backup check on healthchecks.io went green again.

The lesson: test the way it will actually run

Both jobs passed their tests. Both tests ran in a different environment from the real one:

Job	How it was tested	How it really runs
Heartbeat	One manual ping, on a good connection	Every 5 minutes, all day, over a flaky route
Backup	Started from an interactive shell	Started by systemd at 03:30, with no HOME

If I'd found the backup bug in month two instead of on day one, I'd have had weeks of "backup failed" alerts to deal with from abroad, or worse, I'd have missed them. Before I leave:

• Run every unit with systemctl start, not by running the script by hand. That's the environment the timer gives it.
• Do a full reboot test, so everything starts from cold the way it will after a power cut.
• Make alerts mean one thing. One problem should raise one alert, from the system best placed to notice it.

Two pages on day one is annoying. Two pages in month two, from another country, would have been much worse.