drift: scheduled rendered-vs-deployed Caddyfile watcher (Phase 1)

erwin commented

2026-07-09 03:10:01 +00:00

First-time contributor

The watch whose absence caused the familie incident. Schedule (6h) + dispatch, NO apply path: render both Caddyfiles from the checked-out ref, slurp deployed, diff, classify CLEAN/CHANGED/REMOVALS/ERROR (REMOVALS outranks all), post to ntfy homelab-cd, run goes red unless CLEAN.

Merge after milton is converged (the runner's ssh reach). Forgejo only runs schedules from the default branch — first heartbeat within 6h of merge, or fire a dispatch.

The watch whose absence caused the familie incident. Schedule (6h) + dispatch, NO apply path: render both Caddyfiles from the checked-out ref, slurp deployed, diff, classify CLEAN/CHANGED/REMOVALS/ERROR (REMOVALS outranks all), post to ntfy homelab-cd, run goes red unless CLEAN. Merge after milton is converged (the runner's ssh reach). Forgejo only runs schedules from the default branch — first heartbeat within 6h of merge, or fire a dispatch.

erwin added 1 commit

2026-07-09 03:10:01 +00:00

drift: scheduled rendered-vs-deployed Caddyfile watcher (Phase 1) 12cb3bb50f

Watch-only layer, built deliberately BEFORE any apply pipeline. On
2026-06-28 a branch-only deploy of familie.gierz.eu was silently
amputated by a later converge from main and nothing noticed; the fix
starts with detection, not automation. This adds:

- playbooks/render-configs.yml — renders both Caddyfiles from the
  working tree with connection:local, no facts, no SSH. vars_files
  loads caddy + workshop + unified_portal role defaults so the inline
  template fallbacks never fire and the render matches a real converge.
- scripts/drift-check.sh — slurps deployed /etc/caddy/Caddyfile per
  host through the repo inventory (connection semantics stay
  inventory-driven), diffs against the render, classifies
  CLEAN/CHANGED/REMOVALS/ERROR. REMOVALS means a converge from this
  tree would delete deployed lines — the familie failure mode. Also
  counts failed systemd units per host for the heartbeat.
- .forgejo/workflows/drift.yml — every 6h + on demand, on the
  self-hosted runner (gierz-services). Vault password arrives as an
  Actions secret written to RUNNER_TEMP under umask 077 and removed
  in an always() step; ANSIBLE_VAULT_PASSWORD_FILE overrides the
  workstation-only .vault_pass.sh. Verdict goes to ntfy
  (homelab-cd on ntfy.findorff.gierz.eu) FIRST, then the run turns
  red on anything but CLEAN — ntfy being down cannot hide dirtiness,
  and the Actions list shows drift even if nobody reads the phone.

The bot identity is one env knob (CD_BOT_NAME=milton); renaming it is
a one-line change. There is no apply path here and none should be
added to this workflow — applies come later, gated on the
diff-before-apply rule this detector establishes.

Signed-off-by: erwin <erwin@gierz.eu>
Co-Authored-By: LLM <noreply@invalid>
Claude-Session: https://claude.ai/code/session_01UVBwAUedQi9YJ6S9UJomMS

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin erwin/drift-detection:erwin/drift-detection

git switch erwin/drift-detection

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff erwin/drift-detection

git switch erwin/drift-detection

git rebase main

git switch main

git merge --ff-only erwin/drift-detection

git switch erwin/drift-detection

git rebase main

git switch main

git merge --no-ff erwin/drift-detection

git switch main

git merge --squash erwin/drift-detection

git switch main

git merge --ff-only erwin/drift-detection

git switch main

git merge erwin/drift-detection

git push origin main

Rows
Columns

drift: scheduled rendered-vs-deployed Caddyfile watcher (Phase 1) #7

Checkout

Merge