Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Agent Eval Format

Status: design as of P4.3. Crate (forthcoming impl): yanshi-agents::eval + CLI yanshi eval run. Implements: P4.4 (runner + CLI).

The agent eval harness measures whether the agent produced the right answer with acceptable cost. It is intentionally not a unit test framework — eval cases describe end-to-end behaviour (“given this prompt, the agent must call the web_search tool at least once and the final answer must mention ‘OAuth’”) and run a live agent loop end to end against the dataset.

It is a sibling of the existing RAG eval harness (yanshi-rag::eval) and reuses its on-disk style (JSONL + a small TOML manifest) so the two halves of “is the Yanshi stack working?” feel like one tool to operators.

When to use

  • Release gate: pin agent behaviour before tagging a version. P4.4 blocks release-gate quality claims on a green eval run.
  • Regression catch: a refactor of ReActAgent, the reflection strategy, or memory layering should not silently change which tools the agent picks.
  • Skill verification: a skill author runs yanshi eval run on the skill’s bundled fixtures before publishing.

It is not suitable for:

  • Provider quality benchmarks (docs/LLM_PROVIDERS_MATRIX.md and P3.6 are the right surfaces).
  • Retrieval quality (docs/RAG_EVAL.md).
  • Latency profiling under load (P7.1 benches).

Dataset layout

One dataset = one directory.

my_eval_dataset/
  dataset.toml           # name, version, source, license, default limits
  cases.jsonl            # one EvalCase per line
  fixtures/              # optional: files referenced by cases
    invoice_001.pdf
    customer_export.csv

The format intentionally mirrors yanshi-rag/eval_datasets/: JSONL keeps the file diff-friendly and append-friendly, and a manifest TOML lets the dataset declare defaults that every case inherits.

dataset.toml

schema_version = 1
name = "rust-expert-smoke"
version = "0.1.0"
description = "End-to-end smoke for the rust_expert skill."
source = "internal"
license = "CC0-1.0"

# Per-case defaults. Any case can override.
[defaults]
skill = "examples/skills/rust_expert"
max_steps = 12
max_tool_calls = 6
cost_limit_usd = 0.50
latency_limit_ms = 60000
model = "mock-model"

defaults is optional. When omitted, every case must specify the fields explicitly.

cases.jsonl

One EvalCase per line:

{
  "id": "rust-expert-001",
  "prompt": "Review this unsafe block: `unsafe { ptr.read() }`",
  "skill": "examples/skills/rust_expert",          // override
  "tools_allowed": ["shell", "file"],              // narrows admission
  "max_steps": 8,
  "max_tool_calls": 4,
  "cost_limit_usd": 0.10,
  "latency_limit_ms": 30000,
  "model": "mock-model",
  "inputs": { "code_path": "fixtures/sample.rs" }, // optional, injected
  "expected_assertions": [
    { "type": "contains", "needle": "ptr.read" },
    { "type": "tool_called", "tool": "file", "min_count": 1 },
    { "type": "tool_not_called", "tool": "shell" },
    { "type": "step_count_below", "max_steps": 6 }
  ],
  "notes": "Reproduces the 2026-04 customer report."
}

EvalCase fields

FieldTypeRequiredDefault sourceNotes
idstringyes—Stable identifier; appears in reports + trace IDs.
promptstringyes—The user input that opens the run.
skillstringconditionaldefaults.skillPath to the skill dir or registry name.
tools_allowedstring[]noinherit from skillSame precedence as --allow-tool CLI flag (P3.5). Narrows the registry the runtime sees.
tools_deniedstring[]no[]Mirrors --deny-tool for explicit refusal.
inputsobjectno{}Top-level key/value injected into the agent’s initial state.
max_stepsintnodefaults.max_steps → RuntimeLimits::default().max_steps (15)Maps directly to RuntimeLimits::max_steps.
max_tool_callsintnodefaults.max_tool_callsMaps to RuntimeLimits::max_tool_calls.
cost_limit_usdfloatnodefaults.cost_limit_usdHard cap on accumulated provider cost. Run fails with CostLimitExceeded when crossed.
latency_limit_msintnodefaults.latency_limit_msMaps to RuntimeLimits::timeout_ms.
modelstringnodefaults.model or skill manifestProvider model id; honours the same lookup as yanshi workflow run --model.
expected_assertionsarrayyes—See below. Empty array is a hard error (always-pass cases are a mistake).
notesstringno—Free-form human note. Surfaced in failure reports.

schema_version (top-level inside each JSONL line) is optional and defaults to 1. Future incompatible changes bump this.

Assertion DSL

Each entry in expected_assertions is a tagged JSON object. Six variants today; the closed set is part of the v1 stability promise.

contains

{ "type": "contains", "needle": "OAuth", "in": "final_answer" }
FieldDefaultMeaning
needle—Substring to find (case-sensitive).
in"final_answer"Where to search: final_answer / any_step / any_tool_result.
case_insensitivefalseLowercase both sides before matching.

regex

{ "type": "regex", "pattern": "(?i)\\boauth\\s+token\\b", "in": "final_answer" }
FieldDefaultMeaning
pattern—Rust-flavour regex (regex crate).
in"final_answer"Same options as contains.

tool_called

{ "type": "tool_called", "tool": "web_search", "min_count": 1, "max_count": 3 }
FieldDefaultMeaning
tool—Tool name as registered in the agent’s ToolRegistry.
min_count1Minimum invocations.
max_countusize::MAXMaximum invocations. Use to forbid retries past N.
with_paramsabsentOptional JSON object subset — every key must match the call’s params.

tool_not_called

{ "type": "tool_not_called", "tool": "shell" }

Strict variant of tool_called with max_count = 0. Surfaced separately because the typical failure mode is “agent called the forbidden tool” and the report message reads better when the assertion type names it.

step_count_below

{ "type": "step_count_below", "max_steps": 8 }

Asserts the run terminated with strictly fewer than max_steps AgentSteps. Useful when a refactor accidentally inflates the loop without hitting RuntimeLimits::max_steps.

final_answer_matches_skill

{ "type": "final_answer_matches_skill" }

Runs the skill’s bundled fixture verifier (tests/smoke.sh or the skill manifest’s [validation] section) against the final answer. Lets skill authors keep a single authoritative pass/fail function without duplicating it into every eval case.

If the skill has no validator declared, this assertion fails with AssertionUnsupported { reason: "skill declares no validator" }.

Output schema

yanshi eval run <dataset> produces one report per invocation. Default output is human-readable; --format json emits the machine-readable envelope.

JSON envelope

{
  "schema_version": 1,
  "dataset": "rust-expert-smoke",
  "dataset_version": "0.1.0",
  "started_at": "2026-05-15T10:12:00Z",
  "finished_at": "2026-05-15T10:13:42Z",
  "summary": {
    "total": 12,
    "passed": 11,
    "failed": 1,
    "skipped": 0,
    "cost_usd_total": 0.83,
    "latency_ms_p50": 4200,
    "latency_ms_p95": 12100
  },
  "cases": [
    {
      "id": "rust-expert-001",
      "status": "passed",
      "trace_id": "01HZ3K…",
      "started_at": "…",
      "finished_at": "…",
      "duration_ms": 3812,
      "cost_usd_actual": 0.04,
      "stop_reason": "final_answer",
      "step_count": 5,
      "tool_call_count": 2,
      "assertions": [
        { "type": "contains", "passed": true },
        { "type": "tool_called", "tool": "file", "passed": true, "actual_count": 2 }
      ]
    },
    {
      "id": "rust-expert-006",
      "status": "failed",
      "trace_id": "01HZ3K…",
      "duration_ms": 21034,
      "cost_usd_actual": 0.12,
      "stop_reason": "max_tool_calls",
      "step_count": 9,
      "tool_call_count": 6,
      "assertions": [
        {
          "type": "tool_not_called",
          "tool": "shell",
          "passed": false,
          "actual_count": 1,
          "reason": "shell called once at step 4"
        }
      ],
      "notes": "Agent fell back to shell after web_search 503'd; check VCR fixture."
    }
  ]
}

Status values

StatusWhen
passedEvery assertion passed AND stop_reason.is_success().
failedOne or more assertions failed, OR the run terminated with a non-success stop reason (MaxSteps, MaxToolCalls, Timeout, TokenBudgetExceeded, CostLimitExceeded, Error, Cancelled).
skippedCase was filtered out (--filter, --skill, manifest gate, etc.). Skipped cases do not count toward pass/fail.

stop_reason values map 1:1 to AgentStopReason variants (see yanshi-agent-spi/src/runtime.rs) plus the new CostLimitExceeded value introduced by the eval harness when cost_limit_usd is crossed.

Cross-reference with trace replay

Every failed case carries a trace_id that resolves through the existing yanshi trace replay machinery:

yanshi eval run my_dataset --format json > report.json
# Pick a failed case from the report:
jq -r '.cases[] | select(.status=="failed") | .trace_id' report.json | \
  xargs -n1 yanshi trace replay

The eval runner writes traces under the same YANSHI_TRACE_DIR the rest of the stack uses, so trace replay works without any extra flag plumbing. For TUI debugging, swap replay for tui.

Reusing Flow as the eval pipeline

The runner is internally a Flow (yanshi-core::Flow) with one node per case. This is a deliberate choice:

  • The concurrency budget is the same as everywhere else: Flow’s FlowExecutionMode::Concurrent with max_concurrency = N runs cases in parallel.
  • Checkpoints land in the normal ~/.yanshi/checkpoints/<run_id> tree so a long eval can be resumed via yanshi workflow resume-plan <eval_run_id>.
  • Trace events emit through the existing EventListener chain, so --otel flags Just Work.
  • The Flow itself can be inspected with yanshi workflow validate.

Concretely, the runner constructs:

Flow {
  nodes: [
    EvalCaseNode { id: "rust-expert-001", ... },
    EvalCaseNode { id: "rust-expert-002", ... },
    …
  ],
  mode: Concurrent { max_concurrency: cli.parallelism },
}

EvalCaseNode::execute builds a ReActAgent (or the runtime named in dataset.toml), threads RuntimeLimits from the case fields, runs the agent to terminal state, evaluates assertions against the captured AgentSteps + final answer, and emits a per-case JSON row.

CLI surface (lands under P4.4)

# Run a dataset and print a human-readable summary
yanshi eval run path/to/dataset

# Machine-readable JSON envelope
yanshi eval run path/to/dataset --format json --output report.json

# Filter by case id glob
yanshi eval run path/to/dataset --filter "rust-expert-*"

# Force a parallelism for local debugging
yanshi eval run path/to/dataset --parallelism 1

# Treat eval failure as a hard CI signal
yanshi eval run path/to/dataset --fail-on-status failed

# Compare against a checked-in baseline (mirrors `rag eval --compare-baseline`)
yanshi eval run path/to/dataset --compare-baseline baselines/main.json

# Regenerate the baseline file after a deliberate change (T2.1)
yanshi eval run path/to/dataset --dump-baseline baselines/main.json

--compare-baseline/--dump-baseline are implemented (T2.1) via yanshi_agents::eval::{EvalBaseline, compare_against_baseline}. The baseline JSON is three summary numbers computed over non-skipped cases — success_rate, avg_step_count, avg_tool_call_count — not a per-case diff. Comparison is regression-only tolerance checking (a metric moving in the favorable direction never fails the gate), not RAG eval’s paired significance test: a smoke-scale dataset (a handful of cases) doesn’t have RAG eval’s per-query granularity to test significance against. .github/workflows/quality.yml::agent-eval-smoke runs this against yanshi-agents/eval_datasets/ci_offline and yanshi-agents/eval_baselines/ci_offline/baseline.json on every PR. --compare-baseline and --dump-baseline are mutually exclusive.

Exit codes:

  • 0 — every case passed (or skipped).
  • 1 — one or more cases failed, but the dataset shape was valid.
  • 2 — dataset or assertion config was invalid; no agent run was attempted.

Determinism and the mock provider

CI runs the eval harness against the mock provider by default. The mock provider replays canned responses, so a green eval is a strict contract that the agent loop, memory layer, and tool dispatcher haven’t drifted. Live provider runs are opt-in via YANSHI_LIVE_PROVIDER=1 and gated on the relevant API key — matching the P3.6 pattern.

Reproducibility checklist for case authors:

  • Pin the model id (model = "mock-model" in dataset.toml [defaults]).
  • Seed any randomness in tools (e.g. web_search mocks should honour YANSHI_MOCK_SEED).
  • Provide every external input through inputs or fixtures/, never via env vars.
  • Avoid Timeout-based assertions for non-mock runs; use latency_limit_ms only for budget enforcement, not for behaviour.

Stability

  • EvalCase JSON fields, dataset.toml [defaults] keys: stable at first land under P4.4.
  • Assertion DSL closed enum of six variants: stable at first land. Future variants must come through a schema_version bump.
  • JSON report envelope shape: stable; additive fields are tolerated per the P0.3 stability tier.

See docs/STABILITY.md for tier definitions.

  • docs/RAG_EVAL.md — the sibling harness this format mirrors.
  • docs/AGENT_RUNTIME.md — AgentStep, AgentStopReason, RuntimeLimits reference.
  • docs/STABILITY.md — what “stable” means for the assertion DSL.
  • yanshi-agent-spi/src/runtime.rs — source of truth for AgentStopReason and RuntimeLimits.
  • yanshi-rag/src/eval/ — implementation reference for the JSONL + manifest + runner pattern.