ci: converge pipeline with rendered-vs-deployed diff gate (Phase 2) #8

Open

erwin wants to merge 4 commits from erwin/converge-pipeline into main

erwin commented

2026-07-09 03:10:03 +00:00

Collaborator

MERGE LAST, EYES OPEN: merging this arms converge-on-push — the merge commit itself triggers the first gated converge of all:!workstations.

Plan-on-PR (fails on removals, diff to ntfy), converge-on-push (diff posted BEFORE apply; removals block unless workflow_dispatch acknowledge_removals=true — the interim for Forgejo 15's missing environments), workflow-level concurrency, post-apply endpoint verification (fail-closed), vault password lifetime = one step (trap).

Prerequisites: milton merged AND converged; secrets ANSIBLE_VAULT_PASSWORD + NTFY_CD_TOKEN (done); branch protection requiring the plan check (Settings → Branches); checkout action resolvable. Supersedes feature/gitops-cd — close it after this merges.

MERGE LAST, EYES OPEN: merging this arms converge-on-push — the merge commit itself triggers the first gated converge of all:!workstations. Plan-on-PR (fails on removals, diff to ntfy), converge-on-push (diff posted BEFORE apply; removals block unless workflow_dispatch acknowledge_removals=true — the interim for Forgejo 15's missing environments), workflow-level concurrency, post-apply endpoint verification (fail-closed), vault password lifetime = one step (trap). Prerequisites: milton merged AND converged; secrets ANSIBLE_VAULT_PASSWORD + NTFY_CD_TOKEN (done); branch protection requiring the plan check (Settings → Branches); checkout action resolvable. Supersedes feature/gitops-cd — close it after this merges.

erwin added 1 commit

2026-07-09 03:10:03 +00:00

ci: converge pipeline with rendered-vs-deployed diff gate

converge / plan (pull_request) Failing after 30s

Details

converge / converge (pull_request) Has been skipped

Details

b5d245eebd

Supersedes the unmerged feature/gitops-cd branch. Its good bones are
kept - the homelab-converge concurrency group (at WORKFLOW level:
Forgejo honours concurrency only there, and without a group the
platform default cancels an in-flight run on a new push - a converge
killed mid-apply is exactly the partially-converged-fleet failure this
pipeline exists to prevent), the all:!workstations scope,
workflow_dispatch with a limit input - but plan/converge no longer run
blind.

Why: on 2026-06-28 a branch-only live deploy of familie.gierz.eu was
silently amputated by a later converge from main and nothing noticed.
Two rules now hold structurally in the workflow:

1. Never deploy ahead of main. push triggers are filtered to main and
   the converge job hard-fails any workflow_dispatch on another ref.
2. No converge applies blind. Both jobs run ansible in check+diff
   mode first, parse the unified-diff hunks out of the output with one
   shared parser (scripts/extract_check_diff.py - hunk line counts
   from the @@ headers, so content lines starting with "-" cannot be
   misread; unparseable output fails closed), and post the diff to
   ntfy topic homelab-cd before anything else happens. On PRs,
   removals of deployed lines fail the check with the removed lines
   printed in the log. On converge, the apply step sits after the ntfy
   post (a failed post aborts the job) and removals stop the run red
   with an urgent notification unless explicitly acknowledged.

Removal acknowledgement is the workflow_dispatch boolean input
acknowledge_removals, derived from the event name so a plain push can
never satisfy it. This is interim: the preferred mechanism, a Forgejo
protected environment with required approval, does not exist yet -
verified 2026-07-08 against code.gierz.xyz (Forgejo 15.0.1) and the
docs, where jobs.<job_id>.environment appears only in the context
table with no protection semantics. Revisit when Forgejo grows
environments.

Guard rails beyond the gate: dispatch --limit always gets
:!workstations appended (gierz-laptop is ansible_connection: local -
a stray "all" would converge the laptop stack on the runner itself);
post-apply endpoint verification runs under set -e so a broken URL
extraction fails the step instead of reporting success; the vault
password is written, used and removed inside each step that needs it
(trap EXIT), with the always() cleanup as backstop; the PR plan's ntfy
post is advisory so an ntfy outage cannot redden a clean check, while
the converge pre-apply post stays a hard gate by design.

Requires (documented in the header): erwin/milton-runner converged
first, repo secrets ANSIBLE_VAULT_PASSWORD + NTFY_CD_TOKEN, checkout
action resolvable, branch protection requiring the plan check.

Signed-off-by: erwin <erwin@gierz.eu>
Co-Authored-By: LLM <noreply@invalid>
Claude-Session: https://claude.ai/code/session_01UVBwAUedQi9YJ6S9UJomMS

paul added 1 commit

2026-07-26 07:10:31 +00:00

Merge branch 'main' into erwin/converge-pipeline

converge / plan (pull_request) Failing after 48s

Details

converge / converge (pull_request) Has been skipped

Details

221715717f

paul added 1 commit

2026-07-26 07:57:19 +00:00

plan-mode fixes: host identity + check-mode compatibility across site.yml

converge / plan (pull_request) Failing after 42s

Details

converge / converge (pull_request) Has been skipped

Details

1954fd35c2

Plan run 98 findings, verified by iterated local --check sweeps
(edge and gateway fully green; services green to the garage gate):

- host_primary_user (group_vars): the HUMAN owner of a host, paul on
  server groups, connecting-user on workstations. Replaces
  ansible_user_id in common (XDG), libvirt_host (libvirt group +
  image cache), landing_page (docroot owner) — CI converges as the
  bot must not build /home/milton trees or grant milton VM control.
- landing_page: skip build+rsync LOUDLY when the controller has no
  sites checkout (CI runner case); docroots left untouched.
- check-mode compatibility: guard get_url downloads (cloud images,
  grafana dashboards — HEAD 404s), let dotfiles' read-only forge
  probes run under check, restart handlers skip in plans (new-unit
  units don't exist yet in check mode).

Not fixed here: garage asserts /dev/vdc which was never tofu-applied
— main is not convergeable until that disk exists or garage leaves
services.yml. Paul's call.

Co-Authored-By: LLM <noreply@invalid>
Claude-Session: https://claude.ai/code/session_0116pK9Y6ZxhC2S8zFfpFW9i

paul added 1 commit

2026-07-26 08:18:27 +00:00

check-mode compatibility: fleet-wide guard sweep (80 files, scripted)

converge / plan (pull_request) Failing after 2m38s

Details

converge / converge (pull_request) Has been skipped

Details

cdab62f091

Plan run 100 hit adguardhome on the hypervisor — same missing-unit-
under---check species as blackbox before it. Rather than per-run
whack-a-mole, one scripted idempotent pass over every role:

- handlers: systemd restarts gain 'when: not ansible_check_mode' —
  a plan restarts nothing; the triggering diff tells the story.
- tasks: systemd enable+start gains failed_when tolerating ONLY
  missing-unit-under-check (units the play itself renders). Real
  runs stay strict; a stopped existing unit still fails a plan.

Verified: syntax across all playbooks; local --check green on edge +
gateway, services green to the garage /dev/vdc gate (real debt, not
an artifact — needs tofu apply or garage's removal from services.yml).

Co-Authored-By: LLM <noreply@invalid>
Claude-Session: https://claude.ai/code/session_0116pK9Y6ZxhC2S8zFfpFW9i