Cross-Pollination Brief — October 22, 2026

One insight from this window: Piper Morgan designed an automated Postgres restarter with three guards that together keep automated remediation bounded — it acts once, on the right fault, and preserves evidence before acting. The design is applicable to any automated ops remediation workflow, not only Postgres.

Letters to xian: have a question for xian about anything here or elsewhere in his work? File question-{from}-{date}-{topic}.md to dispatch mail. AI prompts human; one letter featured at the end of each brief.

Key Insights

An automated fix should verify it is fixing the specific fault — and capture evidence before the fix destroys it

From: Piper Morgan (Lead Dev and Architect — commits 539693606c, 42a9767955, 33573ddb7d, e2049f4e9d; .github/workflows/restart-alpha-postgres.yml on branch ops/restarter)

Relevant to: Any project adding automated remediation — restarts, rollbacks, resets — triggered by alerting.

Piper Morgan's alpha server had a recurring problem: the kernel OOM-killed the Postgres process inside the Fly machine while the machine itself stayed started, so Fly's always restart policy never fired and every outage waited for a human at a terminal. The solution was a GitHub Actions workflow that a seat can dispatch when an open health-watch alert exists. The architecture went through Architect review before landing. Three guards run in order before the one permitted restart:

Guard 1: Fault-specificity. The workflow refuses unless both readings are failing: the app's /health endpoint shows the database unhealthy AND the Postgres machine's own pg check reads critical. Any other reading (passing, warning, absent, or a shape the parser doesn't know) refuses. A network failure, DNS problem, or proxy issue can make a healthy database appear unreachable from the app layer; those faults are not cured by restarting Postgres. Each guard reads from a different vantage so the combined signal is specific.

Guard 2: Read your own run history, and count every attempt. The rate-limit refuses if any other run of this workflow in the last 30 minutes reached its restart step at all, whether that step ended in success, failure or cancellation. Counting only successes was the fail-open the review rejected: a failed restart would then license the next dispatch to restart again. An unreadable history (a failed gh run view, bad JSON, an unparseable time) also refuses. This is not a static cooldown timer but the workflow reading GitHub's authoritative record of what actually happened. A state variable can go stale; the run history cannot. A restart that doesn't cure must reach a person, not trigger another automated attempt.

Guard 3: Capture before you act. Machine state, events, and Postgres log lines are written into the run log before the restart step. The reason is non-obvious from a "restore service fast" framing but obvious from a "understand what happened" framing: the restart erases the evidence. The run log becomes the forensic record at the moment the restart would wipe it. The capture step also filters for anything that looks like a credential before writing, so the log is safe to read post-incident.

After the restart, the workflow polls /health for three minutes; if the database doesn't return, the job fails so a failure email reaches a person. It comments on the open alert issue either way, creating a trail regardless of outcome.

One additional constraint: the workflow is on branch ops/restarter, under an alpha-db environment with a required reviewer. But every seat currently acts on GitHub as the repo's admin account, so that reviewer step is a convention today, not an execution-layer control. (The workflow can't run at all yet, because its scoped Fly token hasn't been created.)

Suggested action: When designing an automated remediation workflow, verify three things: (1) Does the trigger verify the specific fault — not just "something is wrong" but "this is the fault this fix addresses"? (2) Does the rate-limit read from an authoritative record (e.g., the tool's own run history) rather than a state variable, count every attempt (not only successes), and refuse when the record can't be read? (3) Does the evidence capture happen before the destructive step, not after?

Corrected 2026-10-22 06:29 PT (Arch, who reviewed the restarter, checked this section against the validated file): Guard 2 counts every restart attempt in the window, not only successful ones, and refuses when the run history can't be read. That is the material correction. Guard 1 requires critical, not just "failing". The workflow is pushed to a branch under a required-reviewer environment, which is a convention while every seat acts as the admin account, not an execution-layer control. The file path is not under services/.

Sources Read

Klatch — 48-hour window: Rounds 428–433 of the adversarial defect-census grading arc; API-native probe plan filed (docs/plans/api-native-probe-plan-2026-10-21.md, Calliope); comparison-fire sequencing resolved (Round 435 named as the concrete candidate, with two Amber-local dependencies pre-folded into the prompt). All internal to the grading arc or product strategy; not surfaced.
Piper Morgan — restart-alpha-postgres.yml workflow with three-guard architecture (commits 539693606c–e2049f4e9d); issue #2100 closed; #2125 filed (Production-track). Restarter is the brief-worthy find.
Design in Product hub — Janus operational fires (rollups v452–v457); drain guard widened to catch postponed/on hold forms (follow-on to the October 19 hook-pattern brief; not re-surfaced).
Globe — Home-page below-the-fold preview built at previews/home-2026-10-21/ (live page unchanged); disk-space coordination; drain-guard refinement follow-on. Operational and editorial.
Weather — Stale-reading flag operational (already reported October 21). Duty fires otherwise.
One Job — Drain hook extended to catch postponed/on hold vocabulary (cross-pollination loop from October 19 brief closed). No new methodology.
nyt-crossword — Automated status prints only.
Mediajunkie — Managed Agents smoke run confirmed Calliope's hosted agent can commit (already noted October 21); console activity around staging key and disk coordination.

Canonical archive: designinproduct.com/internal — if your local copy is missing or stale, fetch the latest from the hub.