Monitoring
- Live runs · latency · cost · tokens
- Errors & failure rate (30d)
- By agent, model, and tool
Observe every run
See, score, and prove the quality of every agent run.
Cortex Observability watches every governed run end to end — monitoring, per-run quality scoring, eval release gates, a simulation lab, a reliability score that gates production publish, an improvement queue, a full-trace agent debugger, an event stream, and notifications. Live runs return quality { overall: 100 } — and the numbers trace back to the run that earned them.
Per-run scoring · eval gates · reliability gate · full replay
Observability
latency (ms)
Faithfulness0.98
Citation cov.0.95
Safety1.00
promptcontextmodeltoolsapprovaloutput
ai-insights ▸ quality { overall: 100 }
The problem
You can't govern what you can't measure.
Most teams ship agents on vibes — a green demo, a hopeful prompt, and no idea whether quality held when it reached real work. Without per-run scoring, eval gates, and a reliability bar on publish, a regression goes live silently and you find out from the customer, not the dashboard. Observability turns agent quality into a number you can watch, gate on, and put in front of an auditor.
Reliability is a gate, not a chart — a low-score agent is refused at publish409 RELIABILITY_TOO_LOW
The full observability stack
Nine ways to watch, judge, and improve every run.
One toolkit over the signals Cortex already collects — from a single run trace to a fleet-wide reliability bar that decides what ships.
AI insights
- Per-run quality scoring
- Faithfulness · citation coverage
- quality { overall: 100 }
Evaluation
- Gold eval suites as release gates
- Pass-rate feeds the score
- Block a regression before launch
Simulation lab
- A/B prompts & models
- What-if on real traffic
- Compare before you promote
Reliability score
- 0–100, gates publish (409)
- Weighted: success · eval · incidents
- Trend snapshots over time
Improvement queue
- Low-scoring runs surfaced
- Triage → fix → re-evaluate
- Close the quality loop
Agent debugger
- Full run trace, hop by hop
- prompt → context → model → tools
- Replay any historical run
Event stream & alerts
- Live event feed of every run
- Notifications on threshold breach
- Wire to incidents & oversight
How it works
From a single run to a number that gates production.
The reliability score is a read-model over signals Cortex already records — so it costs nothing extra to collect, and never blocks ops when scoring is unavailable.
- 01
Score every run
Each governed run is scored for quality — faithfulness, citation coverage, safety — and the trace is captured for replay. quality { overall: 100 } means the answer held against its source.
- 02
Roll up to reliability
Per-agent reliability blends success rate (0.5), eval pass rate (0.3), and incidents (0.2) over a 30-day window into a 0–100 score with a band: excellent · good · fair · poor.
- 03
Gate the publish
The registry refuses to publish an agent that has enough signal and scores below the bar — 409 RELIABILITY_TOO_LOW. New agents publish freely; scoring outages fail open.
The gate with teeth
A reliability bar that actually stops a bad release.
Reliability isn't a vanity metric on a wall chart. agent-registry-service.publish() consults the live runtime score and refuses to publish when an agent has sufficient signal and scores below RELIABILITY_MIN_PUBLISH (default 50). New and low-data agents publish freely; if scoring is unavailable it fails open — availability of scoring never blocks your ops.
- Publish refused 409 RELIABILITY_TOO_LOW when score < the bar
- Sufficient = primary signal + totalRuns ≥ 3 (configurable)
- Trend snapshots on every score — watch quality move, not just a single reading
Prove it — don't just claim it
The demo you can run yourself.
Query reliability for a low-score agent that has enough runs, then try to publish it. The registry refuses with 409 RELIABILITY_TOO_LOW — the same call returns a clean score and band you can hand to a reviewer.
Full-trace debugging
Replay any run, hop by hop.
When a run misbehaves, you don't guess. The agent debugger reconstructs the entire run — prompt, retrieved context, model call, every tool invocation, the approval, and the guarded output — each step stamped with its cost, latency, and verdict. Every hop links to the trust ledger, so the trace you read is the run that actually happened.
- Full timeline: prompt → context → model → tools → approval → output
- Per-hop cost, latency, tokens, and pass/hold/block verdict
- Replayable from history; every step traces to the trust ledger
Works with the whole runtime
Observability sits on top of every gate.
It scores, traces, and gates the same governed runs your agents already flow through — so what you measure is the real thing, and what you prove ties back to the ledger.
Control Tower
- Fleet-wide reliability in the overview
- Avg score · below-threshold count
- Pause a degrading agent in one click
Trust Ledger
- Every run trace recorded
- Hash-chained, tamper-evident
- Scores trace to the runs that earned them
Agent Studio
- Reliability gate on publish
- Chat-test before you ship
- Improvement queue closes the loop
Oversight
- Score drops can trigger tighter modes
- Risk floor it can only tighten
- Notifications wire to incidents
Security & compliance
Built for the enterprise security review.
Tenant-scoped scoring, fail-open availability, tamper-evident run traces, and a reliability gate on publish — mapped to the frameworks your auditors already use.
SOC 2ISO 27001ISO 42001EU AI ActNIST AI RMFFINRA
Measure your agents. Gate what ships.
Score every run, replay any of them, and refuse a bad release before it reaches production — all from one observability plane.