Composable callers and middleware

Harn's agent_loop and harness.llm.call historically exposed only a flat options dict for retry / fallback / shadow / budget behavior. v0.8 opens an explicit caller seam: agent_loop accepts an llm_caller: closure that owns the single harness.llm.call(...) invocation, and the std/llm/* modules ship composable middleware for retry, fallback, shadowing, prompt rewriting, logging, budgeting, caching, circuit breaking, ensembles (best-of-N, self-consistency, debate), prompt refinement, model-aware default packs, and structured-output safety.

Concretely:

• llm_caller(opts?) from std/llm/caller is the blessed default stack — with_retry(default_llm_caller(), opts?.retry ?? {}) with typed reserved-status classification and billed-empty re-dispatch on by default. Reach for it before composing retry by hand.
• default_llm_caller() from std/llm/caller is the bottom of the middleware stack. It mirrors agent_loop's built-in invocation.
• with_* wrappers in std/llm/handlers take a next caller and return a new caller. Compose them left-to-right with compose([...]).
• Multi-call quality strategies (best-of-N, self-consistency, debate) live in std/llm/ensemble.
• Catalog-aware option packs (pack_for, pack_chat, pack_agent, …) live in std/llm/defaults.
• Token / context heuristics live in std/llm/budget.
• Envelope-shaped consolidations (safe_call, safe_field, …) live in std/llm/safe.
• System-prompt builders live in std/llm/prompts.
• Catalog accessors (model_info, resolved_options, family_of, lineage_of, complementary_reviewer, has_capability) live in std/llm/catalog.

Caller contract#

A caller is a closure with the canonical shape:

fn(call) -> LlmCallerResult
//   call = {prompt: string, system: string|nil, opts: dict,
//           turn:  {iteration: int, session_id: string, attempt: int}}
//   LlmCallerResult =
//     {ok: true,  value: <llm dict>}
//   | {ok: false, status: string, error?: any, retryable?: bool}

Reserved statuses:

"budget_exhausted", "transient", "caller_aborted", "caller_skipped", "exception", "schema_validation", "rate_limited", "timeout", "network", "provider_5xx", "stream_interrupt", "context_window_exceeded" (alias "context_overflow"), "auth", "policy_blocked", "circuit_open", and "provider_error" (terminal provider classes — invalid_request / model_unavailable / unknown-terminal — never retried).

Anything else lands in error and is preserved but not interpreted. Wrappers must catch raw thrown errors and re-emit them as {ok: false, status: "exception", error: <raw>} so middleware composition stays total — only the bottom (default_llm_caller) does the throw → envelope translation.

Gotchas#

These bit the implementing agents and will bite users:

1. with_retry does not mutate call.turn.attempt between retries. The original call dict is passed through unchanged. If you need per-attempt counting inside a custom caller, use an atomic — it stays correct even when the caller runs inside parallel/spawn, where concurrent branches share one captured cell (see HARN-LNT-064).
2. compose takes a single list, not varargs. compose([with_logging({}), with_retry({})])(base), not compose(a, b, c)(base).
3. text: "DONE" is a protocol-stripping marker in synthetic test callers. The runtime strips it from visible_text. Use "all-ok" or another string in user-visible synthetic text.
4. with_budget short-circuiting only manifests in agent_loop when loop_until_done: true. Otherwise the loop terminates naturally on iteration 1.
5. refine_prompt's session cache is best-effort. Harn does not support out-of-closure mutation, so threading a mutable session dict requires the caller to do the threading.
6. Streaming is out of scope for llm_caller. harness.llm.stream_call keeps its own surface; a future llm_streaming_caller may parallel this.
7. Off-by-one in retry semantics. The removed llm_retries: 3 historically meant 4 total attempts; with_retry's max_attempts: N means N total attempts. Migrations adjusting llm_retries: K should pass max_attempts: K + 1.

---

std/llm/handlers — composable middleware#

Higher-order functions returning a caller. Each (next, opts) middleware (with_retry, with_logging, with_budget, with_circuit_breaker, with_repair, with_coerce, with_timeout) supports two interchangeable call shapes:

with_retry(next, opts)   // direct: returns a caller
// curried: returns fn(next) -> caller, drops into compose
with_retry(opts)

Compose with compose([with_logging({...}), with_retry({...})])(base) or wrap explicitly with with_retry(default_llm_caller(), {...}). Telemetry goes via agent_emit_event when call.turn.session_id is non-empty.

Minimal example#

import {default_llm_caller} from "std/llm/caller"
import {with_retry, with_logging, compose} from "std/llm/handlers"

const caller = compose([
  with_logging({level: "info"}),
  with_retry({max_attempts: 4, backoff: "exponential"}),
])(default_llm_caller())

const result = agent_loop(harness, task, system, {
  loop_until_done: true,
  llm_caller: caller,
})

Model-option resolution#

For role/env model resolution, use agent_model_options from std/agent/options (the pre-0.10 std/agent/stack bundle was removed; compose retry/logging/budget middleware explicitly with the handlers above and tool middleware from std/llm/tool_middleware):

import {agent_model_options} from "std/agent/options"
import {default_llm_caller} from "std/llm/caller"
import {compose, with_logging, with_retry} from "std/llm/handlers"

const route = agent_model_options({
  role: "planner",
  defaults: {
    provider: "anthropic",
    model: "claude-sonnet-5",
    reasoning_task: "agent",
  },
})
const middleware = [with_retry({max_attempts: 3}), with_logging({})]
const caller = compose(middleware)(default_llm_caller())
const result = agent_loop(
  harness, task, system,
  route.options + {loop_until_done: true, llm_caller: caller},
)

agent_model_options(config?) resolves explicit options first, then role environment prefixes (HARN_AGENT_PLANNER_*, HARN_LLM_PLANNER_*, HARN_PLANNER_*), then shared HARN_AGENT_* / HARN_LLM_*, then defaults. If a model is present it calls pack_for(...), resolves tool_format: "auto", and strips unsupported reasoning or prompt-cache fields before the call reaches a provider (agent_sanitize_model_options exposes the stripping step directly).

Persona-shaped example: cost moat substrate#

The full handler stack is the cost moat substrate for the Opinionated Harn Stack: cheap-model-by-default with frontier escalation only on ambiguity, deterministic budget enforcement per persona, and receipt-grade structured logs for every model call.

import {default_llm_caller} from "std/llm/caller"
import {
  with_retry, with_logging, with_budget, with_routing, with_fallback,
  with_circuit_breaker, compose,
} from "std/llm/handlers"

// Cheap default: a fast / inexpensive model on a tight retry budget.
const cheap = with_circuit_breaker(
  with_retry(default_llm_caller(), {max_attempts: 2}),
  {threshold: 5, reset_ms: 30000},
)

// Frontier escalation: a stronger model with longer retries + a fallback
// to a second strong model if the first provider trips a circuit.
const frontier = with_circuit_breaker(
  with_fallback([
    with_retry(default_llm_caller(), {max_attempts: 3}),
    with_retry(default_llm_caller(), {max_attempts: 2}),
  ]),
  {threshold: 5, reset_ms: 30000},
)

const receipts_sink = { record ->
  // Forward to cloud receipts / IDE-host transcript / etc.
  agent_emit_event("ops.receipts", "llm_call_log", record)
}

// with_routing is a base caller (it owns the call, not a wrapper around
// `next`); the budget + logging middleware compose over it.
const router = with_routing({
  default: cheap,
  routes: [
    {name: "frontier",
     when: { call ->
       call?.opts?.reasoning_task == "judge"
         || (call?.opts?.escalate ?? false)
     },
     caller: frontier},
  ],
})

const persona_caller = compose([
  with_logging({level: "info", sink: receipts_sink}),
  with_budget({max_total_tokens: 250000, max_calls: 200}),
])(router)

agent_loop(harness, task, system, {
  loop_until_done: true,
  llm_caller: persona_caller,
})

with_routing chooses cheap-vs-frontier before the request goes out (so cost stays predictable); with_budget enforces the persona's USD/token cap deterministically; with_logging's sink emits receipt records consumed by a cloud platform's ops console.

First-class routing_policy (recommended)#

The handler composition above is the right tool when each route needs arbitrary closure-based custom logic. For the much more common "failover chain plus per-call budget" case, build a routing_policy once and pass it to harness.llm.call(... routing: policy ...) directly:

const policy = routing_policy({
  chain: [
    {provider: "anthropic", model: "claude-opus-4-20250514"},
    {provider: "openai",    model: "gpt-5.4-mini"},
    {provider: "ollama",    model: "llama4:70b"},      // local fallback
  ],
  failover: {
    on_status: [429, 500, 502, 503, 504],
    on_timeout_ms: 30000,
    on_error_kinds: ["rate_limit", "schema_validation"],
    max_attempts: 3,
  },
  latency: {race_after_ms: 5000},
  budget:  {per_call_usd: 0.5, on_exceed: "abort"},
  observe: {emit_event: "billing.routing_decision"},
  // optional verifier chain
  escalate_on: [
    {kind: "typecheck"},
    {kind: "lint", forbidden_patterns: ["TODO"], on_fail: "refine"},
  ],
})

const result = harness.llm.call(
  "Summarize this PR.", nil, {routing: policy},
)

escalate_on makes frontier escalation conditional on a code-quality signal rather than static routing. Verifiers see the candidate's text after a successful link: accept keeps it, refine retries the same link with the verifier's reason appended to the prompt (capped by max_refines_per_link, default 1), escalate advances to the next link. Three built-in kinds — typecheck (harn-parser), lint (regex forbidden_patterns / required_patterns / max_line_length), and test_run (spawns a configurable command with the candidate on stdin) — let scripts gate "only call Opus when Devstral's answer doesn't typecheck." See the quickref for the full per-kind option list.

Compared to compose([with_routing, with_retry, with_fallback]), the primitive is replay-deterministic (every attempt rides on the result envelope's routing block), records its own tape events (<dispatch>.{decision,attempt,race_started,race_won,race_lost,budget_exceeded,verifier_signal,exhausted}), and pays for latency.race_after_ms racing out-of-the-box. Migrate existing chains by replacing the wrapper composition with one routing_policy({...}) call; layer compose([with_logging, with_budget, ...]) over the policy only when you need bespoke closure-level instrumentation.

Error / envelope semantics#

Every wrapper returns the same envelope shape:

• {ok: true, value: <llm dict>} on success
• {ok: false, status: <reserved string>, error?: any, retryable?: bool} on failure

Wrappers must not throw. with_retry and with_fallback decorate their results with retries_attempted / fallback_index so callers can observe what happened. The agent_loop runtime validates the shape at the seam and emits a friendly diagnostic if a custom caller returns a non-dict.

Composition story#

The agent_loop seam (opts.llm_caller) accepts any caller. For direct harness.llm.call-style usage, callers can invoke their wrapped caller explicitly:

const caller = with_retry(default_llm_caller(), {max_attempts: 3})
const envelope = caller({
  prompt: "hello",
  system: nil,
  opts: {provider: "auto", model: "claude-sonnet-5"},
  turn: {iteration: 0, session_id: "", attempt: 1},
})
if envelope.ok { harness.stdio.log(envelope.value.text) }

std/async.retry_with_backoff is not the same surface — it operates on arbitrary closures with a predicate, not on the caller seam. Use with_retry for llm_caller middleware.

---

std/llm/ensemble — multi-call quality strategies#

Citations (in source):

• Self-consistency: Wang et al. 2022, arxiv:2203.11171.
• Multi-agent debate: Du et al. 2023, arxiv:2305.14325.

Minimal example#

import {best_of_n} from "std/llm/ensemble"

const result = best_of_n(
  "Write a haiku about debugging.",
  "You are a poet.",
  {n: 5, sampler_opts: {temperature: 1.0}},
)
harness.stdio.log(result.best.text)
harness.stdio.log(result.reasoning)

Composition with agent_loop#

best_of_n returns a result dict, not a caller. To use ensemble sampling inside agent_loop, wrap it in a caller:

const ensemble_caller = { call ->
  const r = best_of_n(call.prompt, call.system, {n: 3} + call.opts)
  if !r.ok { return {ok: false, status: r.status} }
  return {ok: true, value: {text: r.best.text}}
}

agent_loop(harness, task, system, {llm_caller: ensemble_caller})

Ensemble functions emit ensemble_cost events with estimated token-fanout cost. Wrap in with_budget to cap aggregate spend.

---

std/llm/refine — meta-prompt prompt rewriting#

target_size: "auto" heuristic over estimate_text_tokens(user_prompt, model):

• <= 60 → "small"
• <= 1200 → "medium"
• otherwise → "large"

The refiner asks the model to emit a single DIFF: <summary> trailer which is parsed off the response. Citations: DSPy MIPROv2, OpenAI Prompt Optimizer guide, OpenAI Cookbook meta-prompting recipe.

Example#

import {refine_prompt} from "std/llm/refine"

const r = refine_prompt({
  user_prompt: "summarize this report",
  style: "imperative",
  target_size: "small",
  keep: ["MUST cite section numbers"],
})
harness.stdio.log(r.refined)
harness.stdio.log(r.diff_summary)

The session cache is best-effort: pass opts.session = {...} and re-pass the same dict on subsequent calls to short-circuit identical refinements.

---

std/llm/budget — token and context heuristics#

Example#

import {
  recommend_max_output_tokens, fits_in_context,
} from "std/llm/budget"

const max_out = recommend_max_output_tokens({
  prompt: long_text,
  system: sys,
  model: "claude-sonnet-5",
  task_kind: "summarize",
})

if !fits_in_context(long_text, "gpt-5.4-mini") {
  // compress / summarize first
}

---

std/llm/defaults — model-aware option packs#

pack_for(opts) returns a complete harness.llm.call-ready options dict resolved through the runtime catalog and capability matrix, then pinned to a task. User options always win.

Layering (low → high):

1. resolved_options(opts) — runtime catalog defaults
2. task overlay (only fills unset fields)
3. harness.llm.apply_reasoning_policy(...) — capability-owned lowering
4. recommend_max_output_tokens(...) when a prompt has no output limit
5. canonical user options — highest precedence

Edge cases#

• Routes with only adaptive thinking: capability lowering selects the adaptive wire shape instead of inventing a manual token budget.
• Ollama Qwen3 + reasoning_policy: "off": the runtime performs the capability-driven /no_think lowering; the pack does not duplicate it.
• provider: "auto" unresolvable: minimal pack.

Example#

import {pack_agent} from "std/llm/defaults"

const opts = pack_agent("claude-sonnet-5", {
  reasoning_policy: "high",
  reasoning_scale: "large",
})
agent_loop(harness, task, system, opts + {loop_until_done: true})

---

std/llm/safe — DRY consolidations#

Example#

import {
  safe_call, safe_field, with_case_insensitive_keys,
} from "std/llm/safe"

const r = safe_call(
  prompt, system, {provider: "auto", model: "gpt-5.4-mini"},
)
if !r.ok { return r }

const envelope = with_case_insensitive_keys(parse_json(r.value.text))
const verdict = safe_field(
  envelope, ["verdict", "decision", "result"], "unknown",
)

---

std/llm/prompts — system-prompt builders#

Example#

import {system_prelude} from "std/llm/prompts"

const sys = system_prelude({
  persona: "You are a release auditor.",
  tone: "terse",
  constraints: ["Cite evidence by file path", "No speculation"],
  output_contract: {
    format: "json", required: ["risks", "recommendation"],
  },
})

---

std/llm/catalog — Harn-side catalog accessors#

Thin wrappers over the llm_resolved_options / llm_model_info Rust builtins. The Harn-side names are deliberately shorter and don't shadow the builtins.

Example#

import {
  complementary_reviewer, family_of, has_capability, named_model_ladder,
} from "std/llm/catalog"

if has_capability(model, "thinking") {
  // This route can honor a provider-neutral reasoning policy.
}

const fam = family_of(model)       // e.g. "anthropic-claude"
const routes = named_model_ladder("agent_frontier").steps
const reviewer = complementary_reviewer({
  author_model: model, intent: "plan_review",
})

---

See also#

• LLM calls — harness.llm.call, harness.llm.call_structured, harness.llm.call_safe.
• Agent loops — agent_loop and the llm_caller: seam.
• LLM quick reference — the one-page cheat sheet.