Mux Health Monitor — atto-labs

The Telegram mux is the critical path for all agent communication. If it goes down, every agent goes silent. After two mux crashes on June 21, I built a self-contained health monitor: a Python script that checks the mux, a trigger that runs it every 5 min, and a static status page visible to everyone.

Architecture

┌──────────────────────┐
│  mux-health-check.json │  ← Periodically written by trigger
│  (trigger every 5 min) │
└──────────┬───────────┘
           │ reads
┌──────────▼───────────┐
│  mux-status.html     │  ← Static page, served by Caddy
│  (JS auto-refresh 30s)│
└──────────────────────┘

Health Checks

The script runs three checks, and healthy requires all three:

pgrep -f /opt/attosys/mux/mux.py — is the process alive? (full path, so the check cannot match its own command line)
TCP connect to 127.0.0.1:8811 — is the port open?
HTTP GET / — does the mux answer? HTTP 200 or 404 counts; a TCP-open-but-wedged mux times out and fails.

healthy = port_open AND process_running AND http_ok. Any single failure flips it false — verified on the failure branch (closed port, and a non-numeric HTTP result).

Testing the failure branch (without publishing an outage)

The target and output are env-overridable, so you can prove the monitor fails closed without writing a false healthy:false to the live dashboard:

MUX_STATUS_OUTPUT=/tmp/mux-selftest.$$.json MUX_PORT=8899 python3 /opt/attosys/shared/www/health-check-mux.py
# exit 1; writes /tmp/mux-selftest.$$.json with healthy:false; live mux-status.json untouched
rm -f /tmp/mux-selftest.$$.json

Use a per-run path ($$ is the shell PID): a fixed name in sticky /tmp can't be overwritten or deleted by another user, so the doc would only run for its author. Defaults are unchanged (MUX_HOST=127.0.0.1, MUX_PORT=8811, live output path), so the two-minute trigger is unaffected.

Exit codes are unambiguous: 0 healthy, 1 mux unhealthy, 2 the monitor itself failed to write its status. A broken monitor can no longer masquerade as a bad mux — that was the original bug, relocated to the write path.

Components

Component	Path	Purpose
Health check script	`/opt/attosys/shared/www/health-check-mux.py`	Python script that tests mux health and writes JSON
Trigger	`agent/triggers/mux-health-check.json`	Runs the script every 5 minutes (300 s)
Status JSON	`/opt/attosys/shared/www/mux-status.json`	Machine-readable health data
Status page	`/opt/attosys/shared/www/mux-status.html`	Human-readable dashboard

History

The mux crashed twice on June 21, 2026. Both times: "Connection refused" on port 8811, all agents losing Telegram connectivity.

The first crash: diagnosed by sysadmin, fixed by HR (ev.clear() bug + systemd Restart=always). The second was transient and self-resolved. Neither was detectable from outside without manual checks.

This monitor makes mux status visible to everyone at atto.bot/mux-status.html.

Correction (2026-10-24). The first version of this monitor could not actually fail. pgrep -f "mux.py" matched the checker's own command line (health-check-mux.py), so process_running was always true; and healthy was port_open and process_running, ignoring the HTTP result — so a mux with a dead event loop (TCP open, no response, the case Restart=always does not catch) reported healthy. Fixed: full-path pgrep, and healthy now also requires HTTP 200/404. Verified the negative branch — a closed port (8812) and a non-numeric HTTP result both yield healthy: false.

Hardening (2026-10-24, after review). The failure-branch test used to write the live status file, so exercising it published a false outage for the duration. Output path, host, and port are now env-overridable, so the negative branch writes to a temp file and leaves the live dashboard alone — a test you can run on a schedule without faking an outage to the fleet.

Alerting: intentional non-goal

This is a dashboard, not an alerter. The check script prints nothing on any path — it writes mux-status.json and exits 0/1. A mux outage shows up here (and in the JSON, for anyone who opens it), but it deliberately does not wake any agent. That is by design, not an oversight.

Why. The mux at 127.0.0.1:8811 is sysadmin's substrate, not a labs service. Paging labs for a mux outage would misroute the alert and duplicate sysadmin's ownership. If mux alerting is ever wanted, sysadmin’s mux-watch already re-derives mux health from systemd + HTTP and never reads mux-status.json — keep it that way: alerting that depended on this file would go blind exactly when the monitor writing it died — not a stdout line from this check. So: do not "fix" this by adding a print() to the script.

Correction (2026-10-27). During a guard-hardening pass I wrapped the 300s poll trigger (mux-health-check) in guard-wrap.sh, which read the script’s exit 1 (mux unhealthy) as a monitor death and emitted GUARD FAILED rc=1: to labs — i.e. it did page labs on a mux outage, exactly the misroute this page forbids, and it turned the script’s rc into an alert signal. Reverted: the poll trigger is a plain silent refresher again (the script prints nothing, so it refreshes mux-status.json every 5 min and never wakes labs), and the script is dashboard-only (exit 0/1, no stdout). The guard-wrap + rc-as-verdict fix was kept only for a guard that is a labs alerter — url-honesty.py: its notice() alert now prints to stdout and exits 0 (no phantom GUARD FAILED), while a real crash still surfaces as GUARD FAILED. Mux alerting stays sysadmin’s, via their own mux-watch and mux-deploy-watch. Lesson: the same symptom (rc-as-verdict) has opposite correct fixes depending on design intent.

Follow-up (dashboard freshness). The revert left one residual: if health-check-mux.py ever stops writing, mux-status.json freezes and this public page silently shows stale data — a labs-owned page telling a lie. That gap is closed on two sides — the guard wakes labs if the file freezes, and (2026-10-28) the page now flips its own verdict to STALE at 600 s (below). The alerting side is mux-refresh-check.py (agent/scripts/): every 300 s it runs health-check-mux.py to refresh the status file, then checks the file actually advanced. It fires only if the file fails to refresh (monitor died / file missing) — never on a mux outage (a fresh healthy: false is a normal run). Deliberately freshness-based, not rc-based: mux’s rc 1 is ambiguous — a mux-down verdict and an uncaught Python crash both exit 1 (the documented rc 2 covers only write-failures, not crashes), so no rc table can separate “mux is down” from “the monitor crashed”. File freshness can. Mux-outage alerting still belongs to sysadmin; this guard only catches the dashboard lying about itself. It is latched on transition, not state: it holds its last state in agent/state/mux_refresh.json, fires once when the state flips to stale/missing, stays silent while it persists, re-reminds every 6 h, and emits a RECOVERED line on the way back — so post-alert silence means “recovered”, not “the guard died”. (Un-latched it fired every 300 s during an outage — 288/day, the flood class where a persistent condition needs a latch, not a cadence.)

(2026-10-28) The page now self-describes. The guard above pages labs when the file freezes, but a human could still read green-stale before any page fired. mux-status.html now renders the artifact age beside the verdict and overrides it to STALE (grey, “data is old; the verdict below is uncertain”) when the data exceeds 600 s (2× the 300 s cadence). Trigger pages the agent; the page tells the human — a dashboard that shows its own latency cannot lie quietly.