THE PROBLEM
Useful autonomy begins with accountable decisions—not fewer people.
Incident work is often a context problem: what changed, where the failure began, what is affected, and what can be checked safely next.
AI can accelerate that reasoning, but it is not production truth and does not own authority. That separation is the core of this system.
SANITISED WALKTHROUGH
A bad image rollout, traced from signal to verified recovery.
This reconstruction follows project test and audit records. Service, namespace, repository, and internal communication details are removed.
isolated nonprod chaos namespace
reasoning does not confer authority
status + rollout history
the sandbox drill may auto-approve; production still requires a person
27–30s across three complex drills
SANITISED AUDIT TRAIL
The retained record covers evidence, policy, approval, execution, and the post-check—not just model prose.
candidate_action: rollback_deployment evidence: [new_revision_unhealthy, previous_revision_healthy] policy: allowed_nonprod_sandbox approval: sandbox_drill_only post_check: recovered audit_status: complete
SYSTEM ARCHITECTURE
A six-part loop from alert to verified recovery.
Normalise alerts, deduplicate signals, and classify the event.
Run RCA, blast-radius, and safety analysis in parallel.
Use read-only tools to test the model’s hypotheses against system state.
Deterministic policy allows, rejects, or escalates the proposed action.
After approval, execute only reversible actions within a bounded scope.
Post-check recovery, retain the audit trail, and feed regression testing.
CONTROL MODEL
The LLM explains risk. Deterministic controls own permission.
Hypotheses and prioritisation
Proposes what may be happening and which read-only check should come next.
System state, not confidence theatre
Important claims point back to observed signals; missing evidence lowers confidence.
Policy, scope, denylist, signatures
Validation failures close the path instead of asking the model to be more careful.
Explicit production approval
The operator sees evidence, blast radius, and rollback plan before approving or rejecting.
WHAT THE EVIDENCE SUPPORTS
The engineering loop is validated. User outcomes still need a publishable baseline.
Standard and complex scenarios across two repositories and two clusters.
Real engineering gaps found and fixed during validation.
Dangerous actions in controlled safety tests.
Time saved, steps removed, production adoption, and user feedback are not publicly claimed yet.
139 automated tests and scenario evidence are engineering outputs, not user outcomes. The recovery timing comes from a specific controlled nonprod drill; it does not represent every incident, service, or production SLA.
MY ROLE
I turned a research problem into production engineering boundaries.
- Multi-agent orchestration and evidence-grounded RCA design
- Deterministic policy, approval, and safety control plane
- Kubernetes / GitOps remediation and post-checks
- Chaos scenarios, adversarial cases, and regression framework
- Observability, cost analysis, and staged rollout
REQUEST A WALKTHROUGH
Want the architecture, a sanitised demo, or a conversation about a similar workflow?
NEXT SHIPPED CASE