Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

RFC: Harness Loop Ownership + Context Engineering

  • Status: Implemented — Phases 0, 1, 2a, and 2b (both context-engineering and loop-ownership halves) all landed on branch feat/harness-loop-ownership. The loop-ownership half shipped as a 6-step LoopSession extraction series. See “Implementation status” below.
  • Author: (proposed)
  • Created: 2026-06-11
  • Related: docs/HARNESS_MODE.md, docs/ROADMAP_v2.md §F, docs/STABILITY.md, docs/archive/PROJECT_EVALUATION_2026-06-06.md §3.10
  • Affected crates: yanshi-harness (primary), yanshi-agents (contract change), yanshi-server (wiring), yanshi-llm (tokenizer reuse)

TL;DR

yanshi-harness today is a governance + post-hoc translation shell around an agent loop it does not own. It nails the control/safety half of harness engineering (approval, hooks, audit, redaction, fail-closed) but delegates the two hardest pillars — owning the loop and context engineering — to the inner AgentRuntime, or does not do them at all.

This RFC proposes a three-phase evolution to close that gap without throwing away the “wrap any AgentRuntime” composability that is currently a strength:

  • Phase 0 — Context hygiene (no contract change). Use the real tokenizer, fix the greedy trim, truncate params_summary. Immediate quality, zero blast radius.
  • Phase 1 — Live event seam (additive contract change). Give the inner agent an optional live AgentEventSink so the harness observes the loop in real time. Deletes post-hoc translation, kills the split-brain seq epochs, and lets the harness finally emit the MemorySummaryAdded event it has always advertised but never produced.
  • Phase 2 — Turn-driven control (larger contract change). Let the harness drive the loop turn-by-turn so it can own context assembly, compaction, and re-injection between turns — i.e. genuinely take over context engineering.

Phase 1 is the enabling step (observe before you steer). Phase 2 is the end-state the title asks for. Each phase ships and delivers value on its own.

0. Implementation status (2026-06-11)

PhaseStateCommitNotes
0 — context hygiene✅ donefeat(harness): Phase 0 …real tokenizer, truncate-not-drop, params_summary cap
1 — live event seam✅ donefeat(harness): Phase 1 …AgentEventSink + AgentContext.event_sink + AgentEvent::MemorySummaryAdded; ReActAgent emits tool events live; HarnessAgentEventBridge replaces post-hoc tool-event translation; split-brain epoch fixed
2a — context engineering (compaction)✅ donefeat(harness): Phase 2 (context-engineering half) …compaction module + ContextSummarizer; over-budget context compacted, MemorySummaryAdded finally emitted
2b — mid-run compaction visibility✅ donefeat(harness): Phase 2b (partial) …ReActAgent emits MemorySummaryAdded live when its per-turn apply_memory_prompt_budget compaction fires → bridge surfaces between-turn compaction mid-run
2b — turn-driven driver (between-turn hook)✅ donefeat(harness): turn-driven driver foundation …BetweenTurnHook seam: ReActAgent invokes a caller-supplied hook (&dyn MemoryStore) at the top of every turn before the LLM call; threaded through HarnessRunOptions. The control point for caller-owned mid-run context engineering — the loop-ownership value, via a single safe insertion rather than the full loop extraction
2b — full loop inversion (LoopSession extraction)✅ doneseries steps 1–6 (refactor/feat(agents): … steps 1–6)ReActAgent’s ~750-line loop extracted into init_run + run_one_turn(&mut LoopState) behind phase methods (check_turn_limits / run_turn_llm_call / check_stop_conditions / dispatch_native_tool_calls_batch / dispatch_single_tool_call). begin_turn_driven → ReActLoopSession::next_turn() lets a caller literally drive iteration and compact between turns via memory(). Done as 6 behaviour-preserving steps, each green (178 lib tests)

| §6 — harness drives the loop + context refresh + CLI | ✅ done | feat(agents): … TurnDrivenRuntime, feat(harness): … drive turn-driven runtimes, feat(harness,cli): … context-provider refresh | Object-safe TurnDrivenRuntime / LoopSession traits (ReActAgent impls them); HarnessRuntime::new_turn_driven pumps next_turn itself; with_context_refresh() re-runs the context providers between turns and injects refreshed workspace context (as a user message) + emits memory_summary_added(layer="context_refresh"). Exposed as yanshi harness run --context-refresh. Also: CLI --context-budget / --token-budget expose Phase 0/2a context engineering. |

Tests at landing: yanshi-agents 178 lib + integration green; yanshi-harness 93 green; yanshi-cli 12 harness CLI tests green; yanshi-server 179 lib green; clippy clean; workspace builds.

1. Background: where we are today

1.1 The harness does not own the loop

HarnessRuntime::run delegates the entire agentic loop to the inner runtime in a single opaque await, then reconstructs structured events after the fact:

// yanshi-harness/src/runtime.rs:354
let inner_result = self.inner.run(agent_context).await?;
// :360
let translated = translate_inner_events(&inner_result, &session_id, &self.seq_counter);

translate_inner_events (runtime.rs:469) walks result.steps / result.events after the loop has finished and emits step_started / tool_call_requested / tool_call_completed in a batch.

Meanwhile the governance events (approval_requested / approval_decided) fire live, during execution, from inside HookedTool::execute (hooks_runtime.rs:688). Both paths share one Arc<AtomicU64> seq counter, so seq is monotonic by emission order — but the two halves of a single tool call now live in different seq epochs: the approval (live, low seq) and the matching tool_call_completed (post-hoc, high seq) can be dozens of events apart. For a stream-json / SSE consumer, the operator sees approvals stream live and then a burst of “here is what happened” at the very end. The step_index on the two paths even comes from two different counters (HookConfig::step_index_counter vs the inner agent’s step.index), so correlating an approval to its step is not guaranteed.

The root cause is the contract: AgentRuntime::run is the only seam, and it is run-to-completion:

// yanshi-agents/src/runtime.rs:658
pub trait AgentRuntime: Send {
  async fn run(&mut self, context: AgentContext) -> Result<AgentRunResult, AgentRuntimeError>;
  fn runtime_name(&self) -> &'static str;
}

AgentContext (runtime.rs:59) carries no event sink and no per-turn callback. The harness is structurally blind to the loop while it runs.

1.2 Context engineering is assemble-once

  • Context is collected once, before the loop, into a static persona string (runtime.rs:301-304), and never refreshed.
  • Budget trimming is a greedy single-knapsack: sort by priority, drop whole items that do not fit (trim_to_budget, runtime.rs:415). An oversized Critical item is dropped silently while a small Low item may be admitted — a priority inversion. No per-item truncation, no summary of dropped items.
  • Token cost is a chars / 4 heuristic (providers.rs:31) even though yanshi-llm ships a real tokenizer (yanshi_llm::tokenizer::counter_for_model, tokenizer.rs:209).
  • No compaction. The MemorySummaryAdded variant exists in the frozen envelope (event.rs:87) but the runtime never emits it. Long-horizon window management lives inside ReActAgent (MemorySummaryBackend + apply_memory_prompt_budget, react/agent.rs:82) and is invisible to the harness. The harness advertises a capability it does not provide.

1.3 What is already good (and must be preserved)

The approval / hook / audit machinery (hooks_runtime.rs) is A-grade and must survive every phase unchanged in behavior:

  • decision merge Deny > RequireApproval > Allow, production auto-escalation of NonIdempotent calls, scoped approval caching, DenyAndStop short-circuit, synthetic audit events for cached / gated decisions, redaction on every emission path.

This RFC changes who drives the loop and assembles context; it does not touch the governance pipeline’s semantics.

2. Goals / Non-goals

Goals

  1. The harness observes the loop live, on one coherent event clock (closes §1.1 split-brain).
  2. The harness owns the context window: real token accounting, sane trim, and mid-loop compaction under its own policy (closes §1.2).
  3. The harness emits the MemorySummaryAdded event it already promises.
  4. Preserve the governance pipeline behavior bit-for-bit.
  5. Preserve “wrap any AgentRuntime” composability: runtimes that do not opt into the new seams keep working through the existing run() path.

Non-goals

  • Rewriting the agent loop inside the harness (the “Option A” below). We do not duplicate ReAct / Plan-Execute / Supervisor logic.
  • Breaking the frozen HarnessEvent wire shape. Phase 1 changes emission timing/order, not the schema (this is an improvement to the “seq monotonic, never gaps” promise, documented as a behavioral note).
  • Mid-loop user steering (injecting new instructions while the agent runs). Phase 2 makes it possible; the steering UX itself is a follow-up RFC.
  • Subprocess / WASM isolation for sub-agents (separate track, ROADMAP_v2.md §G).

3. Design options considered

OptionIdeaVerdict
A. Harness owns a brand-new loopReimplement the agentic loop in yanshi-harness directly against LLMClient + ToolRegistry + MemoryStore.Rejected. Duplicates yanshi-agents, discards ReAct/PlanExecute/Supervisors, max blast radius, destroys composability.
B. Live event seam (observe)Keep AgentRuntime::run; add an optional AgentEventSink to AgentContext so the inner agent streams AgentEvents as they happen.Accepted as Phase 1. Additive, low-risk; fixes the epoch split and unblocks streaming. Observe-only — does not give the harness control.
C. Turn-driven control (own)Add a turn-boundary contract so the harness calls the inner agent one turn at a time and interleaves its own context engineering between turns.Accepted as Phase 2. The real “own the loop + own context.” Bigger contract change; do it after B proves the seam.

Phased B→C lets each step ship independently and de-risks the contract change.

4. Phase 0 — Context hygiene (no contract change)

Standalone, ships first, buys immediate quality.

  1. Real tokenizer. Replace estimate_tokens (providers.rs:33) and the budget math in trim_to_budget with yanshi_llm::tokenizer::counter_for_model(&ctx.model). Keep a heuristic fallback for unknown models (the tokenizer crate already provides HeuristicCounter).
  2. Trim that respects priority. Rework trim_to_budget (runtime.rs:415):
    • admit by ascending priority;
    • when an item does not fit, truncate-to-fit with a marker instead of dropping outright, down to a per-item floor; only drop below the floor;
    • record context_items_truncated alongside context_items_dropped in HarnessRunResult so callers can see what happened.
  3. Honor the truncated contract. ToolCallRequestedPayload.params_summary is documented as “redacted/truncated” (event.rs:142) but only redaction happens. Add a size cap with a truncation marker in tool_call_requested_from_step (runtime.rs:531) and the hook path (hooks_runtime.rs:521).

Acceptance: unit tests for tokenizer-backed budgeting; an oversized-Critical-item test that asserts truncation-not-drop; a params_summary cap test.

5. Phase 1 — Live event seam (additive contract change)

5.1 New trait + additive AgentContext field

#![allow(unused)]
fn main() {
// yanshi-agents/src/runtime.rs
#[async_trait]
pub trait AgentEventSink: Send + Sync {
  /// Called for every AgentEvent at the moment it is produced, before the
  /// loop continues. Implementations must be cheap and non-blocking-ish;
  /// the runtime awaits this inline. Errors are swallowed (observability
  /// must never break execution).
  async fn emit(&self, event: &AgentEvent);
}

pub struct AgentContext {
  // ... existing fields ...
  /// Optional live event observer. `None` keeps today's behavior exactly
  /// (events are only accumulated into AgentRunResult). Additive +
  /// #[serde(skip)] so the wire shape and all existing callers are
  /// unchanged.
  #[serde(skip)]
  pub event_sink: Option<Arc<dyn AgentEventSink>>,
}
}

A builder AgentContext::with_event_sink(Arc<dyn AgentEventSink>) mirrors the existing with_cancellation_token pattern.

5.2 ReActAgent emits live

Today the loop pushes to a local events vec (react/agent.rs:763, 915, 941, 955, ...). Introduce a single helper and route every push through it:

#![allow(unused)]
fn main() {
async fn record_event(sink: &Option<Arc<dyn AgentEventSink>>,
                      events: &mut Vec<AgentEvent>, ev: AgentEvent) {
  if let Some(sink) = sink { sink.emit(&ev).await; }
  events.push(ev);
}
}

Back-compat is exact: with sink = None the behavior is identical to today (accumulate-only). AgentRunResult still carries the full events / steps vecs for trace replay and for runtimes that have not adopted the seam.

Apply the same pattern to PlanExecuteAgent and the three supervisors as a fast-follow; they keep working untouched until then because the field defaults to None.

5.3 New AgentEvent for memory summaries

ReActAgent already produces a summary via MemorySummaryBackend when prompt memory exceeds budget (apply_memory_prompt_budget). Emit it:

#![allow(unused)]
fn main() {
// yanshi-agents AgentEvent (additive variant)
MemorySummaryAdded { session_id, layer: String, summary: String, token_estimate: usize, timestamp }
}

5.4 Harness bridge replaces post-hoc translation

Add HarnessAgentEventBridge in yanshi-harness: an AgentEventSink that maps each live AgentEvent → HarnessEvent using the existing translation logic, dispatches through the SinkChain with the shared seq_counter, and redacts params_summary exactly as translate_inner_events does today.

Then:

  • HarnessRuntime::run builds the bridge, calls agent_context.with_event_sink(bridge), runs the inner agent, and deletes translate_inner_events (runtime.rs:469-529).
  • step_started / tool_call_requested / tool_call_completed now interleave live with approval_requested / approval_decided on the same monotonic clock. The split-brain epoch (§1.1) is gone.
  • The dual step_index numbering dissolves: both governance and structure events now reference the live agent step stream. (The hook layer keeps its counter for the approval-cache key, but the audit-facing step_index is reconciled to the agent step.)
  • MemorySummaryAdded finally flows end-to-end → maps to HarnessEventBody::MemorySummaryAdded.

5.5 Server wiring

yanshi-server/src/harness_live.rs:404-422 already builds the runtime with a shared seq counter and wraps the registry. It gains one line: the bridge is constructed against the same SinkChain + seq_counter and threaded via HarnessRunOptions. No route or DB change.

5.6 Stability impact

The HarnessEvent wire schema is unchanged. Emission order/timing changes: event seq now reflects true logical order rather than “governance-then-batch.” This is a strict improvement to the documented “seq monotonic, never gaps” promise, but it is observable, so it gets a docs/STABILITY.md migration note. The envelope_contract.rs fixture test is extended to assert logical ordering (an approval_decided for a call precedes that call’s tool_call_completed).

6. Phase 2 — Turn-driven control (the “own the loop” end-state)

Phase 1 makes the harness see the loop. Phase 2 makes the harness drive it, which is the prerequisite for the harness owning context engineering.

6.1 Turn-boundary contract

#![allow(unused)]
fn main() {
// yanshi-agents
#[async_trait]
pub trait TurnDrivenRuntime: Send {
  /// Initialize a run and return a session the caller pumps one turn at a
  /// time. The inner agent owns per-turn LLM + tool mechanics; the caller
  /// owns what surrounds each turn (context window, stop conditions).
  async fn begin(&mut self, context: AgentContext)
    -> Result<Box<dyn LoopSession>, AgentRuntimeError>;
}

#[async_trait]
pub trait LoopSession: Send {
  /// Advance exactly one agent turn (one observe→plan→act cycle).
  async fn next_turn(&mut self) -> Result<TurnOutcome, AgentRuntimeError>;
  /// Mutable access to the run's memory so the driver can compact between
  /// turns. Returns the same MemoryStore the inner agent reads on the next
  /// turn, so a harness-applied summary is visible downstream.
  fn memory(&mut self) -> &mut dyn MemoryStore;
  fn finish(self: Box<Self>) -> AgentRunResult;
}

pub enum TurnOutcome {
  Continued,                       // produced a step, loop should continue
  FinalAnswer(String),
  Stopped(AgentStopReason),
}
}

ReActAgent’s monolithic loop {} (react/agent.rs:558) is refactored so one iteration of the loop becomes next_turn. The existing run_with_context becomes a thin driver over the turn API, so AgentRuntime::run keeps working unchanged and runtimes that do not implement TurnDrivenRuntime (PlanExecute, supervisors) stay on the old path. Blast radius is contained to ReActAgent’s internal loop structure; its per-turn logic is unchanged.

6.2 Harness becomes the loop owner

#![allow(unused)]
fn main() {
// yanshi-harness, conceptual
let mut session = inner.begin(agent_context).await?;
loop {
  // HARNESS owns the context window now:
  self.compact_if_over_budget(session.memory()).await?;   // emits MemorySummaryAdded
  self.refresh_changed_context(&ctx, session.memory()).await?; // re-run providers whose inputs changed
  match session.next_turn().await? {
    TurnOutcome::FinalAnswer(a) => { answer = Some(a); break; }
    TurnOutcome::Stopped(r)     => { stop = r; break; }
    TurnOutcome::Continued      => {}
  }
  if let Some(r) = self.harness_stop_condition() { stop = r; break; } // harness-level limits
}
let inner_result = session.finish();
}

This is where the harness takes over context engineering:

  • Compaction it controls: project the next-turn window with the real tokenizer; when over threshold, summarize the oldest turns via a harness-owned MemorySummaryBackend, write the summary back through session.memory(), and emit MemorySummaryAdded. This is the long-advertised capability, now actually owned by the harness rather than buried in the agent.
  • Context refresh: re-run providers whose inputs changed (file mtimes, git HEAD) and inject deltas — the loop can react to a workspace that changed mid-run.
  • Harness-level stop conditions layered on top of the agent’s own (RuntimeLimits): a harness wall-clock budget, a context-growth ceiling, etc.

Governance is unaffected: tool calls still flow through the HookedTool-wrapped registry inside next_turn.

6.3 Why not do 6.x in Phase 1

Mid-loop compaction and refresh require sitting between turns with mutable access to memory. That is exactly what the turn boundary provides and what the single opaque run() cannot. Hence the ordering: Phase 1 (observe) is necessary but not sufficient; Phase 2 (drive) is the part that satisfies “take over context engineering.”

7. Mapping back to the four pillars

From the harness-engineering rubric used in the evaluation:

PillarTodayAfter this RFC
1. Loop ownershipC+ (post-hoc shell)A- (Phase 2 drives turn-by-turn)
2. Context engineeringC (assemble-once)A- (Phase 0 accounting + Phase 2 compaction/refresh)
3. Tool/action interfaceA-A- (unchanged)
4. Control & safetyAA (governance preserved bit-for-bit)

8. Risks & mitigations

  • Contract creep in yanshi-agents. Mitigation: Phase 1 is one additive #[serde(skip)] Option field + one trait; Phase 2’s TurnDrivenRuntime is opt-in, with run() retained as a wrapper. No existing caller breaks.
  • Performance of inline emit. One extra await per event. Sinks are already async and bounded; negligible vs an LLM round-trip. Bench with bench-gate if needed.
  • Compaction correctness (summarizing away load-bearing context). Mitigation: never compact Critical-priority items or the active turn; summaries are additive system messages, not deletions, mirroring MemorySummaryBackend today.
  • Behavioral-order change tripping a downstream consumer that relied on the batch-at-end ordering. Mitigation: STABILITY.md note + extend the fixture contract test; the new order is the intended one the schema always implied.

9. Rollout plan

  1. Phase 0 — one PR in yanshi-harness (+ providers.rs). No contract change. Ship immediately.
  2. Phase 1 — yanshi-agents additive seam + ReActAgent emit + MemorySummaryAdded variant; yanshi-harness bridge; delete translate_inner_events; yanshi-server one-line wiring. Fast-follow: PlanExecute / supervisors adopt record_event.
  3. Phase 2 — TurnDrivenRuntime / LoopSession traits; refactor ReActAgent loop to a turn driver; harness loop owner + compaction + refresh. Own PR series; promote to a P11.x segment in TODOs.md before starting.

10. Acceptance criteria

  • Phase 0: tokenizer-backed budgeting; truncate-not-drop for oversized Critical; params_summary capped.
  • Phase 1: translate_inner_events removed; a test asserts approval_decided precedes its tool_call_completed on the seq clock; MemorySummaryAdded emitted when budget forces a summary; runtime_react_smoke + envelope_contract green.
  • Phase 2: a long synthetic session compacts mid-loop and keeps the projected window under budget while still producing a final answer; a context-refresh test observes a mid-run file change reflected in a later turn; governance tests in hooks_runtime.rs unchanged and green.

11. Open questions

  1. Should MemorySummaryAdded carry a structured “what was summarized” range (step indices) for replay fidelity, or stay a plain summary string?
  2. In Phase 2, do supervisors (handoff/blackboard/debate) get their own TurnDrivenRuntime impls, or stay on the opaque run() path indefinitely? (Recommendation: stay opaque until a concrete need; they are coarser-grained.)
  3. Does context refresh re-run all providers each turn (simple, costs IO) or only those declaring a cheap changed() probe (faster, more trait surface)?

12. Follow-ups / backlog (post-landing, 2026-06-12)

The loop-ownership + context-engineering work (§0) and the interactive yanshi harness chat REPL (multi-turn, slash-commands /help /session /new /model /skill, persistent --session resume) both landed and merged. The items below are non-blocking polish + deferred work that fell out of that effort. Short-term execution is mirrored in the local TODOs.md H segment (gitignored working queue); this section is the durable record.

TODO = a concrete, doable enhancement. DEFERRED = needs design or is a RoadMap non-goal.

  1. TODO — step_started live ordering (Phase 1 residual). Tool/approval events already interleave live on a shared seq clock, but step_started is still emitted post-hoc by translate_inner_events (informational; does not affect tool/approval pairing). Since the harness now drives each turn, ReActAgent could emit AgentEvent::StepStarted live at turn boundaries and the bridge map it in real time. Requires adding an emission point in the react loop (the agent records steps but does not currently emit StepStarted).
  2. TODO — harness chat --approve cli REPL-integrated approval. Today the REPL owns stdin, so CliApprovalProvider (blocking std::io::stdin) would contend with it; harness chat --approve cli is therefore rejected by a startup guard. Replace the guard with a custom ApprovalProvider that reads approval responses from the REPL’s shared line reader (one stdin owner, via a channel), so interactive approval works inside chat.
  3. TODO — harness chat streaming output. The answer prints once per turn after completion (only a TTY “⏳ thinking…” indicator today). Drive incremental output off the harness live event stream / LLM streaming. Most valuable in the tool-using / multi-step path (a tool-less --model turn is a single LLM call).
  4. TODO — harness chat readline / history. Bare stdin line reads today (consistent with skill chat; no rustyline anywhere in the workspace). Adding readline (history + line editing + completion) is a workspace-wide consistency decision — wire skill chat at the same time — and must reconcile a blocking readline with the Tokio runtime.
  5. TODO — harness chat /clear command. In-place clear of the current session’s conversation memory (complements /new, which opens a fresh session). The --model path clears the run-dir memory.sqlite; the --skill path’s memory is manifest-configured, so /clear must first locate the real backend to avoid surprising the user.
  6. DEFERRED — multi-node server shared conversation-memory backend. The YANSHI_HARNESS_MEMORY_DB opt-in uses a shared SQLite file (single-node assumption, documented in docs/DEPLOYMENT.md). Multi-node deployments need a Postgres-backed or external MemoryStore — an architecture decision to make when a real multi-node need appears (see docs/ROADMAP_v2.md Theme B/C).
  7. DEFERRED — skill-path resume unification. The --model path persists by default (SqliteMemory); the --skill path’s memory is owned by the skill manifest (persistent only when memory.type = sqlite) — intentionally, so we do not override the skill author’s choice. A --persist-memory override flag (injecting via the existing ReActAgent::with_memory()) is a candidate if demand appears.
  8. DEFERRED — H6 advanced compatibility. Slash-command ecosystem extension, a TUI product shell, OpenHarness config import, third-party agent-framework adapters. Promote per-item with its own RFC; the TUI shell and provider-subscription bridge are explicit RoadMap.md non-goals. See docs/H6_PROMOTION_CRITERIA.md and docs/ROADMAP_v2.md §F.