THE PROBLEM
Engineering knowledge is not a search result. It is context that can be defended.
Real questions cross repositories, architecture notes, ownership, and current infrastructure state. Vector search alone misses exact paths and fresh runtime signals.
RelayOps starts with reproducible prefetch, allows a bounded read-only investigation, then applies citation checks, confidence caps, secret gates, and abstention.
SYSTEM ARCHITECTURE
Retrieval, investigation, and governance each do their proper job.
Use BM25 and a live snapshot to establish reproducible initial context.
Let the agent fill evidence gaps with read-only tools and a bounded budget.
Check each citation, source, and claim for consistency.
Apply confidence caps, abstention, and the secret gate.
Return a traceable answer, visible unknowns, and the next safe action.
SANITISED WALKTHROUGH
One engineering question, from retrieval to a visible unknown.
This example follows evaluation and QA-history fields. Internal repository, path, and service identifiers are removed.
natural language, typo and follow-up tolerant
a reproducible candidate set
bounded budget; no mutation capability
invalid citations are removed; weak evidence triggers a cap or abstention
environment-specific approver mapping remains unknown
Approval occurs after policy validation and before execution. The request includes the proposed action, scope, and rollback plan. Publicly available evidence does not establish the approver mapping for a specific environment, so the service owner must verify that detail.
SOURCES · approval-flow.md · policy-gates.md · live snapshot (read-only)AUDIT TRAIL
Each answer retains the intent, confidence, abstention, sources, tool calls, latency, and token use.
question_id: qa-redacted intent: workflow_lookup confidence: bounded abstain: partial citations_valid: true unknowns: [environment_specific_approver] tools: [prefetch, repository_read, live_snapshot] write_capability: none
SAFETY MODEL
“Read-only” is a capability boundary, not a sentence in the prompt.
No mutating tools
Least-privilege infrastructure access sets a hard ceiling on what the agent can do.
Redaction at several layers
High-risk resources and sensitive tool output are filtered before the final answer.
Claims must remain traceable
The verifier checks that cited evidence appeared in the current investigation.
“I do not know” is a feature
Conflicting, stale, or absent evidence becomes an explicit unknown and safe next step.
WHAT THE EVIDENCE SUPPORTS
Traceability is measurable. User impact still needs a publishable baseline.
Citation validity in the same controlled evaluation set.
False confidence for the hybrid architecture in adversarial cases.
Average tool calls, compared with approximately 8.4 for open exploration.
Time saved, answer adoption, and correct-abstention rate are not publicly claimed yet.
1,678 chunks, 657 files, and 84 tests describe system scale and engineering output—not time saved or adoption. The 0.90 and average tool-call figures describe the same controlled evaluation set.
LEARNING LOOP
Product quality comes from the signals that real questions leave behind.
I reviewed 30 real conversations for gaps around typos, follow-ups, path discovery, and language consistency, then translated the observations into routing, language gates, and regression cases.
That keeps the agent from evolving through prompt tweaks alone: problem categories, replayable cases, and a pre-release quality gate form the engineering loop.
MY ROLE
I owned the system from index and agent through the safety control plane.
- Multi-repository knowledge index and deterministic prefetch
- Bounded agent investigation and read-only infrastructure tools
- Citation verification, confidence caps, and abstention policy
- Prompt-injection fence, secret redaction, and complete audit trail
- Real-conversation analysis, regression testing, and quality iteration
REQUEST A WALKTHROUGH
Want to see citations, abstention, and the audit trail working together?
RELATED SHIPPED CASE