Incident: gierz-services hung 2026-08-08 — workout-tracker bad tag, OOM cascade #12

New issue

Open

opened 2026-08-08 12:39:19 +00:00 by erwin · 1 comment

erwin commented

2026-08-08 12:39:19 +00:00

Collaborator

Incident 2026-08-08 — gierz-services VM hung, all public services down

KB record: knowledge-base:projects/homelab/incidents/2026-08-08-gierz-services-vm-hung

What happened

~2026-08-02 → 2026-08-08 14:00 CEST: gierz-services degraded until fully unresponsive — SSH banner timeout, qemu guest agent dead, code.gierz.xyz / auth.gierz.eu timing out. Recovered 14:21 via virsh shutdown + start (@paul).
Previous boot journal: 524 OOM kills; workout-tracker.service restart counter 25,617 (~every 20 s since Aug 2).

Root cause

roles/workout_tracker/defaults/main.yml:8 pins workout_tracker_image_tag: "v0.27.0" — that tag does not exist for ghcr.io/jovandeginste/workout-tracker (newest real release: v0.11.3). It is a copy-paste of the blackbox_exporter tag from roles/observability/defaults/main.yml:70. Pull fails with "manifest unknown" → infinite crash-loop → memory churn → OOM killer took Authentik workers, then sshd + qemu-ga.

Second incident of this class (see jupyter disk-full, 2026-07-26): a crash-looping Quadlet unit with unbounded restarts degrades the whole host.

Contributing factors

Guest has no swap → memory pressure kills daemons instead of slowing down.
Hypervisor RAM overcommit: 17G vCPU RAM promised on a 14G host, 4.5G into host swap.
Monitoring/alerting lives on the affected host → no alert fired externally.

Interim state (2026-08-08)

workout-tracker stopped + masked.
Load-shedding script ~/personal/homelab/stop-noncore-services.sh: non-core stacks stopped (overleaf, paperless, tandoor, jellyfin, immich leftovers, grocy, homebox, hortusfox, hedgedoc, actualbudget, grafana, loki, headplane, simplex-smp). Reboot/converge resurrects these until encoded.

Fixes (all via repo, per policy)

Pin workout_tracker_image_tag: "v0.11.3" (and audit other roles for copy-pasted tags)
Restart backoff + StartLimitBurst in the shared Quadlet template — no container unit may crash-loop unbounded
Encode the shed services' enabled-state (off for now) in roles/inventory
Swap or zram for gierz-services (roles/server_base)
Reduce hypervisor RAM overcommit (17G on 14G)
External liveness probe independent of gierz-services (edge-based or third-party) + hypervisor memory/load alert
Verify lingering config on hoering-services (gotosocial appeared session-bound)

Filed by @erwin (session 2026-08-08). Assisted-By: @erwin (claude-fable-5)

## Incident 2026-08-08 — gierz-services VM hung, all public services down KB record: `knowledge-base:projects/homelab/incidents/2026-08-08-gierz-services-vm-hung` ### What happened - ~2026-08-02 → 2026-08-08 14:00 CEST: gierz-services degraded until fully unresponsive — SSH banner timeout, qemu guest agent dead, `code.gierz.xyz` / `auth.gierz.eu` timing out. Recovered 14:21 via `virsh shutdown` + `start` (@paul). - Previous boot journal: **524 OOM kills**; `workout-tracker.service` restart counter **25,617** (~every 20 s since Aug 2). ### Root cause `roles/workout_tracker/defaults/main.yml:8` pins `workout_tracker_image_tag: "v0.27.0"` — that tag **does not exist** for `ghcr.io/jovandeginste/workout-tracker` (newest real release: **v0.11.3**). It is a copy-paste of the *blackbox_exporter* tag from `roles/observability/defaults/main.yml:70`. Pull fails with "manifest unknown" → infinite crash-loop → memory churn → OOM killer took Authentik workers, then sshd + qemu-ga. Second incident of this class (see jupyter disk-full, 2026-07-26): a crash-looping Quadlet unit with unbounded restarts degrades the whole host. ### Contributing factors - Guest has **no swap** → memory pressure kills daemons instead of slowing down. - Hypervisor RAM overcommit: 17G vCPU RAM promised on a 14G host, 4.5G into host swap. - Monitoring/alerting lives on the affected host → no alert fired externally. ### Interim state (2026-08-08) - `workout-tracker` stopped + masked. - Load-shedding script `~/personal/homelab/stop-noncore-services.sh`: non-core stacks stopped (overleaf, paperless, tandoor, jellyfin, immich leftovers, grocy, homebox, hortusfox, hedgedoc, actualbudget, grafana, loki, headplane, simplex-smp). **Reboot/converge resurrects these until encoded.** ### Fixes (all via repo, per policy) - [ ] Pin `workout_tracker_image_tag: "v0.11.3"` (and audit other roles for copy-pasted tags) - [ ] Restart backoff + `StartLimitBurst` in the shared Quadlet template — no container unit may crash-loop unbounded - [ ] Encode the shed services' enabled-state (off for now) in roles/inventory - [ ] Swap or zram for gierz-services (`roles/server_base`) - [ ] Reduce hypervisor RAM overcommit (17G on 14G) - [ ] External liveness probe independent of gierz-services (edge-based or third-party) + hypervisor memory/load alert - [ ] Verify lingering config on hoering-services (gotosocial appeared session-bound) --- Filed by @erwin (session 2026-08-08). Assisted-By: @erwin (claude-fable-5)

👀 1

erwin commented

2026-08-08 12:47:48 +00:00

Author

Collaborator

Verification + corrections (erwin, 2026-08-08 ~15:00 CEST)

Correction to "What happened": the ACPI virsh shutdown at 14:12 never completed — the guest was too far gone to process it. Recovery was a hard virsh destroy at 14:20, then start at 14:21 (shell history). Clean boot, no fs damage observed, but noted for the record.
Load-shed script ran 14:37. Verified 15:00: shed units inactive, available memory 682Mi → 2.6Gi, 45 running services (was 68).
actualbudget.service shows failed — it ignored SIGTERM on stop and was SIGKILLed. Down as intended; reset-failed clears the flag. Minor extra datapoint for the Quadlet-template work: stop timeouts deserve a look too.

### Verification + corrections (erwin, 2026-08-08 ~15:00 CEST) - Correction to "What happened": the ACPI `virsh shutdown` at 14:12 never completed — the guest was too far gone to process it. Recovery was a **hard `virsh destroy`** at 14:20, then `start` at 14:21 (shell history). Clean boot, no fs damage observed, but noted for the record. - Load-shed script ran 14:37. Verified 15:00: shed units inactive, **available memory 682Mi → 2.6Gi**, 45 running services (was 68). - `actualbudget.service` shows `failed` — it ignored SIGTERM on stop and was SIGKILLed. Down as intended; `reset-failed` clears the flag. Minor extra datapoint for the Quadlet-template work: stop timeouts deserve a look too.