drift: scheduled rendered-vs-deployed Caddyfile watcher (Phase 1) #7

Merged

paul merged 1 commit from erwin/drift-detection into main

2026-07-09 14:41:13 +00:00

erwin commented

2026-07-09 03:10:01 +00:00

Collaborator

Merge after milton is converged (the runner's ssh reach). Forgejo only runs schedules from the default branch — first heartbeat within 6h of merge, or fire a dispatch.

The watch whose absence caused the familie incident. Schedule (6h) + dispatch, NO apply path: render both Caddyfiles from the checked-out ref, slurp deployed, diff, classify CLEAN/CHANGED/REMOVALS/ERROR (REMOVALS outranks all), post to ntfy homelab-cd, run goes red unless CLEAN. Merge after milton is converged (the runner's ssh reach). Forgejo only runs schedules from the default branch — first heartbeat within 6h of merge, or fire a dispatch.

erwin added 1 commit

2026-07-09 03:10:01 +00:00

drift: scheduled rendered-vs-deployed Caddyfile watcher (Phase 1) 12cb3bb50f

Watch-only layer, built deliberately BEFORE any apply pipeline. On
2026-06-28 a branch-only deploy of familie.gierz.eu was silently
amputated by a later converge from main and nothing noticed; the fix
starts with detection, not automation. This adds:

- playbooks/render-configs.yml — renders both Caddyfiles from the
  working tree with connection:local, no facts, no SSH. vars_files
  loads caddy + workshop + unified_portal role defaults so the inline
  template fallbacks never fire and the render matches a real converge.
- scripts/drift-check.sh — slurps deployed /etc/caddy/Caddyfile per
  host through the repo inventory (connection semantics stay
  inventory-driven), diffs against the render, classifies
  CLEAN/CHANGED/REMOVALS/ERROR. REMOVALS means a converge from this
  tree would delete deployed lines — the familie failure mode. Also
  counts failed systemd units per host for the heartbeat.
- .forgejo/workflows/drift.yml — every 6h + on demand, on the
  self-hosted runner (gierz-services). Vault password arrives as an
  Actions secret written to RUNNER_TEMP under umask 077 and removed
  in an always() step; ANSIBLE_VAULT_PASSWORD_FILE overrides the
  workstation-only .vault_pass.sh. Verdict goes to ntfy
  (homelab-cd on ntfy.findorff.gierz.eu) FIRST, then the run turns
  red on anything but CLEAN — ntfy being down cannot hide dirtiness,
  and the Actions list shows drift even if nobody reads the phone.

The bot identity is one env knob (CD_BOT_NAME=milton); renaming it is
a one-line change. There is no apply path here and none should be
added to this workflow — applies come later, gated on the
diff-before-apply rule this detector establishes.

Signed-off-by: erwin <erwin@gierz.eu>
Co-Authored-By: LLM <noreply@invalid>
Claude-Session: https://claude.ai/code/session_01UVBwAUedQi9YJ6S9UJomMS

paul force-pushed erwin/drift-detection from 12cb3bb50f to e10b581081

2026-07-09 14:38:15 +00:00

Compare

paul merged commit 6220c992a5 into main

2026-07-09 14:41:13 +00:00

paul deleted branch erwin/drift-detection

2026-07-09 14:41:13 +00:00

paul referenced this pull request from a commit

2026-07-09 14:41:13 +00:00

Merge pull request 'drift: scheduled rendered-vs-deployed Caddyfile watcher (Phase 1)' (#7) from erwin/drift-detection into main