observability: watch familie.gierz.eu, alert when any public endpoint dies

erwin commented

2026-07-09 03:09:59 +00:00

First-time contributor

Adds familie to the LIVE probe list (host_vars override — review caught the defaults list being dead code) and a PublicEndpointDown rule (probe_success==0 for 5m, critical → ntfy). Mergeable independently; converge observability whenever.

erwin added 1 commit

2026-07-09 03:09:59 +00:00

observability: watch familie.gierz.eu, alert when any public endpoint dies ea5b4fdaf4

Two gaps let the familie amputation go unnoticed for ten days: the
blackbox exporter never probed familie.gierz.eu, and probe_success was
never alerted on at all - only TLS-expiry rules exist, and those have no
data for a host that is down outright.

- add familie.gierz.eu to the live probe list in
  inventory/host_vars/gierz-services.yml (the observability_scrape_configs
  override there is what the host actually uses) and to the role-defaults
  list, kept in sync and in the same order per the comment in host_vars.
- new PublicEndpointDown rule: probe_success{job="blackbox_http"} == 0
  for 5m, severity critical -> ntfy-critical (X-Priority 5). Covers all
  18 probed endpoints, not just familie.

NOTE: familie.gierz.eu is currently down (edge aborts the TLS handshake;
the incident this CD effort answers). After this converges the alert
will fire truthfully until erwin/familie-caddy-recover is converged.
That is intended - the watch telling the truth.

Signed-off-by: erwin <erwin@gierz.eu>
Co-Authored-By: LLM <noreply@invalid>
Claude-Session: https://claude.ai/code/session_01UVBwAUedQi9YJ6S9UJomMS

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin erwin/observability-probe-familie:erwin/observability-probe-familie

git switch erwin/observability-probe-familie

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff erwin/observability-probe-familie

git switch erwin/observability-probe-familie

git rebase main

git switch main

git merge --ff-only erwin/observability-probe-familie

git switch erwin/observability-probe-familie

git rebase main

git switch main

git merge --no-ff erwin/observability-probe-familie

git switch main

git merge --squash erwin/observability-probe-familie

git switch main

git merge --ff-only erwin/observability-probe-familie

git switch main

git merge erwin/observability-probe-familie

git push origin main

Rows
Columns

observability: watch familie.gierz.eu, alert when any public endpoint dies #6

Checkout

Merge