AI Agent Evaluation and Observability (Part 7): Traces, Graders, and Release Gates

AI Agent Engineering Series

• AI Agents Part 1: Architecture
• AI Agents Part 2: Harness and CLI
• AI Agents Part 3: Tools and APIs
• AI Agents Part 4: MCP
• AI Agents Part 5: Context and Memory
• AI Agents Part 6: Agentic RAG
• AI Agents Part 7: Evals and Observability
• AI Agents Part 8: Security and Deployment

AI agent evaluation means grading what an agent did, and observability is the record that makes that grading possible. Grade the full trace and the final state of the test environment. A correct final message can hide a missing write or an unsafe request. Use code graders for objective checks, repeat each case many times, and let a release gate turn the results into a ship or block decision.

Observability records, evaluation judges

Observability is the ability to see what a run did after it finished. The harness from Part 2 provides it through the trace, the ordered, append-only list of events for one run. Each model decision, tool result, approval, and status change is one event.

Evaluation applies stated criteria to that record and decides if the behavior was acceptable. Anthropic's Demystifying Evals for AI Agents calls the record a transcript, "the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions". OpenAI's trace grading guide defines trace grading as "the process of assigning structured scores or labels to an agent's trace".

The order of work follows from this dependency. Design the trace first, because a grader cannot check a fact that the trace does not record.

The vocabulary below gives one name to each idea. It follows the Anthropic post, with "trace" in place of "transcript".

Grade the outcome and the path

The final message is the agent's claim about what happened. The outcome is what did happen. Anthropic defines the outcome as "the final state in the environment at the end of the trial". Its example is a flight agent. The agent can say the flight is booked, but the outcome is "whether a reservation exists in the environment's SQL database".

A refund agent shows the same gap. In FIG 02, the agent reads the order and the refund policy, then writes a confident final message with a receipt ID. It never calls the refund tool. A test that reads only the message passes this trial.

The two inputs catch different failures. The final state shows what the agent changed. The trace shows how it got there: which tools it asked for, what each returned, and whether an approval came before each write. An agent can reach the right state after an unsafe request that the harness refused. Only the trace records that request.

The trace records model decisions and the tool results that the application can verify. It does not record hidden model reasoning. Keep large or sensitive payloads in protected storage and put a reference in the event. For a full list of what a production trace must capture, see AI Agent Traces and Tool Calls: What to Capture.

Export to OpenTelemetry without moving the source of truth

Many teams send traces to an OpenTelemetry backend. The OpenTelemetry GenAI agent span conventions define an invoke_agent operation and attributes such as gen_ai.operation.name and gen_ai.agent.name. Their status is "Development", so names can change. Keep the event trace as the input for graders and export spans from it. When the convention changes, only the exporter changes.

A trace taxonomy has two halves

The first half is the closed set of event types. These are run started, user turn, context report, model returned, tool or harness observation, compaction, and status changed. Version the set across the series. A closed set lets the eval reject a trace with an event type it does not know. A grader that silently skips unknown events can pass a run it did not understand.

The second half is the set of failure labels. Write them from failed traces, after you read them. Anthropic states the reason: "You won't know if your graders are working well unless you read the transcripts and grades from many trials."

A good label is narrow enough that two reviewers apply it the same way, and it points at the part of the system to fix. "Bad answer" fails both tests. unverified_claim passes both, because a reviewer can check it against the trace and it points at the final-answer step.

One failed trial can trigger several labels. In FIG 02, every check over the trace and the final state fails. During human review, record the first bad event as the cause, because later failures often follow from it. In FIG 02 that is event 5, the final decision.

Choose the grader type by the criterion

There are three kinds of grader. Anthropic recommends "deterministic graders where possible, LLM graders where necessary or for additional flexibility". The choice depends on the criterion, and each type has a different cost and failure mode.

Never use a model grader for approvals, amounts, record counts, or permissions. Code checks those exactly, at no cost.

Five design rules for code graders

1. Check outcomes and invariants. Do not match the exact tool sequence. Anthropic found that step-sequence checks were "too rigid and results in overly brittle tests", because agents find valid paths that the designers did not expect. Check order only where order is the rule, such as approval before a write.
2. Grade the request as well as the execution. A request for a forbidden tool fails, even when the harness refused it. The refusal shows that the harness works. The request shows that the model would cross the boundary under a weaker harness.
3. Check claims against tool results. Every receipt ID or record ID in the final message must appear in a tool result. This catches the trial in FIG 02.
4. Keep graders independent of the code they check. Read the final records in the test services directly, and compare them with the receipts in the trace. That catches a missing, duplicated, or unrecorded write. Do not call the payment service's own logic. A grader that shares a bug with the tool agrees with the wrong answer.
5. Keep criteria separate. Do not combine graders into one weighted score. A correct refund that skipped approval still fails, and no average can hide that failure.

The shape of one grading pass fits in a few lines:

grade(trial, expectations):
  if the trace is empty, out of order, or not finished:
    return [ERROR("trace unusable")]
  verdicts = []
  for each grader in the suite:
    try:
      ok, evidence = grader.check(trace, final_state, expectations)
      verdicts.add(PASS or FAIL, grader.label, evidence)
    on any exception:
      verdicts.add(ERROR, grader.name, the exception)
  return verdicts

Graders are code, so they need their own tests. Build one small trace by hand that must fail each critical grader, and one broken trace that must produce ERROR. A grader with a bug can pass every trial and give false confidence.

ERROR is a separate verdict

ERROR means that the grader, the fixture, or the trace could not support a decision. A truncated trace is the main reason for ERROR. If the trace does not end in a final status, the record is incomplete. A grader that reads it can pass a run that failed. That is an eval problem, so it must not count as an agent failure.

Isolate every trial

A fixture builds fresh fake services and data for each trial. Anthropic's rule is that "each trial should be 'isolated' by starting from a clean environment." Shared state lets one trial change the next. For example, a refund from trial 3 makes the duplicate check fail in trial 4.

Fixtures should use the same tool layer and write path as production, so the eval tests the approval flow that users get. Keep a deliberately weak tool in a fixture when you need to prove that a grader catches a failure. Keep the grader after the tool gains its own check, because a later refactor can remove that check. For fixture design and state capture, see AI Agent Testing: Build a Sandbox Before Production.

One passing trial proves little

The model can take a different path on each run. One pass shows that the agent can succeed once. Three statistics describe a set of trials, and each answers a different question.

The pass@k estimator comes from the Codex paper by Chen et al.. The τ-bench paper by Yao et al. proposed pass^k "to evaluate the reliability of agent behavior over multiple trials". Its results show why the metric matters. Function-calling agents such as gpt-4o succeeded on fewer than 50% of tasks, with pass^8 below 25% in the retail domain.

pass^k falls fast. If each trial passes independently with probability 0.9, all eight pass with probability 0.9^8, about 0.43. At 0.99 per trial, pass^8 is about 0.92. A high pass@k with a low pass^k describes an agent that can do the task but does not do it every time.

The interval shows how little a small run proves

The Wilson score interval gives a range for the true pass rate from a small number of trials. The NIST/SEMATECH handbook describes it and notes that its lower limit cannot be negative, unlike the simple normal approximation. The chart shows the lower bound when every trial passes.

Ten passes out of ten still leave the lower bound at 0.72. Twenty passes raise it to 0.84. A gate that must show a lower bound above 0.95 needs 73 or more clean trials per case.

These counts are starting points. Increase them when the interval is too wide for the decision. For a full method to compare two versions with pinned models, see Harness A/B Testing for Agents.

The release gate

The gate turns verdicts into one decision. It applies two kinds of rule, chosen by the damage a failure causes.

• A critical grader blocks the release on a single FAIL in any trial. Mark a grader critical when one failure is unacceptable: a write without approval, a forbidden tool request, a wrong amount, a duplicate side effect, or an invented receipt.
• Other graders count against a pass-rate floor per case. A missing source citation lowers the pass rate but does not block on its own.

A case that protects a permission boundary gets a floor of 1.0. For example, a refund request for another customer's order must hand off with zero refunds in every trial. Cases like this one become runtime controls in Part 8.

The gate checks for errors before it checks anything else:

gate(results):
  if any trial has an ERROR verdict:
    return 2                     # the eval system is broken
  for each case:
    if any critical grader failed in any trial: block(case)
    if pass_rate(case) < case.floor:            block(case)
  return 1 if any case is blocked, else 0

If a grader is broken, its other results from the same run are not reliable. The gate reports exit code 2 and blocks the release. It still reports any failure that an independent grader confirmed, such as an unauthorized write, while the team repairs the eval.

A pass rate alone hides the worst trial. A case can pass 17 of 20 trials and still contain one invented receipt. That single trial blocks the release, even at an 85% pass rate. Save every failed trace as a CI artifact, so the engineer who reads a block sees the evidence.

Saved traces have a second use. When you add a grader, you can regrade old traces with no new model calls.

Run each layer at the frequency its cost allows

End-to-end trials with a live model are slow and cost money. Most failures can be found earlier and at lower cost.

This order also speeds up diagnosis. If the retrieval check fails, a prompt change cannot fix the missing passage, so no end-to-end run is needed to find the cause.

Anthropic also separates capability evals from regression evals. Capability evals start with a low pass rate and measure progress. Regression evals "should have a nearly 100% pass rate" and protect behavior that already works. Gate releases on regression cases. Track capability cases as a trend, and move a case into the gate when the agent passes it reliably.

Model graders need calibration

Some criteria cannot be code: "the explanation is clear", "the refusal does not reveal private details", "the summary is faithful to the source". A model grader (an LLM judge) is a second model that labels a response against one criterion. OpenAI's evaluation best practices list "Not calibrating your automated metrics against human evals" as an anti-pattern.

A model grader is a measuring instrument with its own error rate. Measure that rate against human labels on the same items before the grader can affect a release. Three numbers matter.

The false-pass rate matters most, because a false pass ships the failure and a false fail only costs review time.

Rules for a model grader:

1. Give it one criterion and a small label set: PASS, FAIL, or UNKNOWN.
2. Give it only the evidence it needs: the request, the response, and the relevant source passages.
3. Do not show it the expected label.
4. Collect human labels on representative items, including hard cases near the boundary.
5. Measure on items that you did not use to write the grader prompt.
6. Version the grader model, prompt, examples, and settings. Measure again after any change.
7. Map every UNKNOWN before the gate counts results. For a critical criterion, treat it as FAIL. For others, send it to human review or count it as ERROR, by a written rule.

Production traces feed the next cases

Offline evals test the failures that you expected. Production traces show the ones you did not. Production writes the same trace format, so the same code graders run on live traces after each run finishes.

Code graders cost almost nothing, so run them on every stored trace. Human review covers a sample, so take more samples from high-risk paths. Group failures by their first failure label to find the component that fails most often.

A trace can contain customer data, order details, and retrieved documents. Redact before storage, restrict access, set a retention period for each data class, and never record secrets or access tokens. Part 8 covers these controls. For the review process that turns incidents into cases, see Every Production Failure Should Become an Eval Case.

Remove cases that no longer protect a decision, because every case costs repeated trials on each release run.

Where this approach stops

Code graders over traces work well when the expected outcome is clear and the user gives one turn of input. Three kinds of work need more.

• Long conversations where the user changes the request need a simulated user that responds to the agent. See Agent Simulation Testing.
• Open-ended output, such as research reports, has mostly subjective criteria. Calibrated model graders and human review do more of the work.
• Low-risk agents with no side effects may not justify a 20-trial gate. A small golden set with human review can cost less than the failures it prevents.

For eval strategy beyond this series, see How to Eval AI Agents in 2026 and Production Evals for Agentic Systems.

Where OpenNash fits

OpenNash forward-deployed engineers build this eval layer with Fortune 2000 teams that run agents with write access. We read your agent's traces with your team and write the failure labels. We build cases and code graders from your workflows, then add the release gate and production sampling. You own the cases, graders, and gate at the end. Book a call with an OpenNash forward-deployed engineer or email hello@opennash.com. In the first session, we review your traces and show which failures cost the most, so the eval work starts there.

Previous: AI Agents Part 6: Agentic RAG

Next: AI Agents Part 8: Security and Deployment turns the critical failure labels into permissions, approvals, isolation, and release controls.