LLM calls

llm_call#

Make a single LLM request. Harn normalizes provider responses into a canonical dict so product code does not need to parse provider-native message shapes.

const result = harness.llm.call("What is 2 + 2?")
harness.stdio.log(result.text)

With a system message:

const result = harness.llm.call(
  "Explain quicksort",
  "You are a computer science teacher. Be concise."
)
harness.stdio.log(result.text)

With options — build them through the typed LlmCallOptions alias from std/llm/options (the documented path; option typos then surface at harn check time instead of being silently ignored):

import { LlmCallOptions } from "std/llm/options"

const opts: LlmCallOptions = {
  provider: "openai",
  model: "gpt-5.4-mini",
  max_tokens: 1024,
}
const result = harness.llm.call(
  "Translate to French: Hello, world",
  "You are a translator.",
  opts,
)
harness.stdio.log(result.text)

With image or video content:

import { image_content, video_content } from "std/llm/media"
import { LlmCallOptions } from "std/llm/options"

const opts: LlmCallOptions = {
  provider: "minimax",
  model: "MiniMax-M3",
  messages: [{
    role: "user",
    content: [
      {type: "text", text: "Summarize these inputs."},
      image_content("diagram.png", {detail: "auto"}),
      video_content("demo.mp4"),
    ],
  }],
}
const result = harness.llm.call("", nil, opts)
harness.stdio.log(result.text)

Image blocks use the provider-neutral shape {type: "image", url?: string, base64?: string, media_type: string, detail?: "low"|"high"|"auto"}. Exactly one of url or base64 is required. Harn translates it to Anthropic source, OpenAI image_url, Gemini inline_data/file_data, or Ollama images fields at the provider boundary. Ollama's REST API only accepts base64 image data, so url image blocks are rejected for provider: "ollama". std/llm/media also provides image_message(...) and image_vision_context(...) helpers when a harness wants the same image as both LLM content and deterministic vision_ocr(...) context.

For Responses requests, images in tool results use the same conversion as ordinary message images. Harn sends their text as function-call outputs and moves visible images into a user message after the consecutive tool results. Malformed base64 tool images, including canonical image blocks without a media type, are omitted while their text is preserved.

Video blocks use the provider-neutral shape {type: "video", url?: string, base64?: string, media_type: string}. Exactly one of url or base64 is required. Harn translates video blocks to OpenAI compatible video_url content, and to Gemini inline_data/file_data parts for routes that declare video support. std/llm/media also provides video_message(...).

Parameters#

When system is given twice#

The system message can arrive positionally (argument 2) or as the system option. Both are accepted, and the outcome depends on which form the option uses:

• Option is a string. The positional argument wins. The option is consulted only when the positional argument is absent, nil, or blank (empty or whitespace only) — then it becomes the system message.
• Option is a fragment list. Nothing competes. The list contributes before / after fragments that surround the primary block, and the positional argument supplies that primary block. Both appear.
• Option is {mode: "replace", content}. The replacement wins over everything, including a positional argument, and suppresses every other prompt contributor.

Passing system both ways is legal but easy to misread. Prefer one form per call site.

Return value#

harness.llm.call returns one rigid, canonical envelope. Every field uses a single snake_case spelling — there are no top-level aliases, and all accounting lives under usage. The typed contract is LlmResponse from std/llm/envelope; the same module exports LlmUsage, LlmOutcome, LlmOutcomeKind, LlmToolCall, and LlmStreamChunk.

The four text channels each have a distinct job — none are aliases. text is the public answer after projection, raw_text is the pre-projection source with protocol tags intact, visible_text is the sanitized human-visible output, and canonical_text is the canonical replay form of a tagged-protocol response.

Effective reasoning effort#

effective_reasoning_effort is always present on a completed call. The Responses API supplies the level through its response's reasoning.effort field. Harn compares that echo with the final request after provider overrides. It doesn't infer a level from the requested effort, thinking budget, or token count.

The record has status: "reported", level, and one of these source values:

Without a usable echo, the record is {status: "not_reported"}. This also applies to older recordings and providers without an effort echo. Responses configuration updates can supersede the echoed request setting. Calls with configuration updates or an opaque previous_response_id therefore return not_reported.

The same observation appears in provider response events, transcript provider_payload metadata, durable assistant messages under _harn.effective_reasoning_effort, and usage.provider_telemetry. Streaming calls include it on the terminal chunk.

Usage#

usage is the single owner of all call accounting; no accounting field is duplicated at the envelope's top level. The VM response, provider-response event, and LLM trace are mechanical projections of the same normalized Rust ledger. Consumers should read the recorded fields rather than re-price tokens or derive cache behavior independently.

Anthropic's wire input_tokens and Bedrock Converse's inputTokens count fresh input only. Harn adds the provider's cache-read and cache-write counts before recording usage.input_tokens. For example, 40 fresh tokens, 5,000 cache reads, and 100 cache writes become 5,140 input tokens. The unmodified counter remains in usage.provider_telemetry.server_prompt_tokens; Anthropic failed-response receipts retain it as reported_input_tokens. OpenAI-compatible prompt totals already include cache and are not increased. Historical Anthropic and Bedrock records produced before this normalization can still contain fresh-only input counts.

harn provider cache-probe --usage-fixture <file> applies the same accounting to saved usage. Raw Anthropic fixtures use cache_read_input_tokens and cache_creation_input_tokens; normalized Harn fixtures use cache_read_tokens and cache_write_tokens with an inclusive input_tokens. Inconsistent normalized counts remain errors. An omitted cache counter remains missing, distinct from a reported zero.

Cache conformance reports use schema version 2. Missing prompt or cache-read evidence yields usage_unreported, including canonical usage whose cache_visibility is undeclared. The report counts those runs and fails the dogfood gate rather than claiming a measured miss. Malformed or conflicting counter aliases yield provider_field_inconsistent. A positive cache read does not require an output or cache-write counter.

For llama.cpp calls, usage.provider_telemetry also separates prompt size from prompt work:

Flat provider_call_response and LLM-trace records use the same canonical names (input_tokens, output_tokens, cache_read_tokens, and cache_write_tokens). The former trace-only cache_tokens mirror is not a separate accounting field.

Provider attempts#

harness.llm.call is one logical call, but the runtime may issue several provider requests to complete it: a rate-limited request is retried, so is one that returns nothing the loop can act on, so is a transport failure. usage.provider_attempts reports that:

Retries are broken out by reason because they mean different things: rate limiting says the provider is saturated, an empty completion says the model produced nothing usable, and a transport retry says the link is flaky.

This is distinct from any counter named for "calls". A run whose provider rejected 47 of 146 requests with retryable 429s still made 96 logical calls, and reporting only the latter hides the contention that explains why the run was slow and stopped early.

Outcome#

Every response carries outcome: {kind, billed}. billed is true when the provider charged tokens for the call. Consumers should branch on outcome.kind rather than re-deriving intent from the provider-native stop_reason:

kind == "empty" together with billed == true is the billed-noncommittal signal — the condition default retry policy re-dispatches on. std/llm/envelope ships predicates for these branches so callers do not re-implement them: llm_response_is_empty, llm_response_is_billed_empty, and llm_response_is_truncated.

Options dict#

LlmCallOptions in std/llm/options is the checked authoring surface. Annotate a binding or use llm_options({...}):

import { LlmCallOptions, llm_options } from "std/llm/options"
import { system_before } from "std/llm/prompts"

const opts: LlmCallOptions = llm_options({
  provider: "openai",
  model: "gpt-5.4",
  system: [system_before("Return a compact result.")],
  output: {
    schema: {
      type: "object",
      properties: {answer: {type: "string"}},
      required: ["answer"],
    },
    strict: true,
    validation: "error",
  },
  effort: "high",
  timeout_ms: 120000,
})

The runtime applies the same registry to typed values, literals, computed dicts, direct calls, streams, and agent-loop dispatch. An unknown key is an error with a nearest-name suggestion. A removed spelling is an error with its replacement. Keys beginning with _ are reserved for internal host plumbing.

Routing#

models, ladder, explicit model/provider, and routing are competing route owners. Do not combine them.

equivalent_failover does not add automatic alternatives for local providers such as ollama, mlx, vllm, llamacpp, or tgi because the catalog does not prove which models are installed. Use an explicit routing policy when those local routes are known to be available.

Use harness.llm.model_ladder(name) to inspect the label and ordered steps of a named ladder without starting a call. Keep reusable provider/model lists in catalog source data and pass only the stable ladder name from policy code.

Conversation#

Each system fragment has {content, title?, position?: "before"|"after", enabled?}. Use system_before, system_after, and with_system_fragments from std/llm/prompts when composing fragments. Use {mode: "replace", content: string} when content must be the entire system channel; replacement suppresses every additive prompt contributor while preserving tools and ordinary conversation history.

Generation#

Harn admits caller-selected temperature, top_p, top_k, seed, frequency_penalty, presence_penalty, stop, and the advanced controls above against the resolved provider/model route before transport. An explicit catalog denial throws a terminal invalid_request error; the option is never silently removed from the wire request. New advanced controls require an authored lowering, including on custom routes. Catalog or provider defaults are not caller intent and do not trigger admission.

Routing applies the same check to every attempted link after that link's model and option overrides are resolved. Run harn provider catalog matrix to inspect the declared route surface before choosing portable options.

The registry records model facts, not merely fields present in a provider's schema. Current Gemini 3.5 and 3.6 routes reject logprobs, and Groq documents its logprobs and logit_bias fields as unsupported by every deployed model. Harn rejects those options before credential lookup. Cerebras accepts logprobs and prediction individually but not together. That combination is also rejected locally.

Output contract and recovery#

output: "json" parses JSON without a schema. A schema value validates and narrows result.data; the config form controls strict provider transport, post-parse validation, and early stream abort in one place:

type Verdict = {pass: bool, reason: string}

const result = harness.llm.call(prompt, nil, {
  output: {
    schema: Verdict,
    strict: true,
    validation: "error",
    stream_abort: true,
  },
  schema_retries: 1,
})

When schema validation remains invalid, the thrown error and harness.llm.call_safe() error include a schema_retry record:

harness.llm.call_structured_result() includes the same record on schema failure. These fields describe corrective schema retries. Provider and network retry policy remains separate.

Reasoning and modalities#

Use effort for intent. Use thinking only when a caller needs to choose the provider mechanism or budget explicitly. Provider capabilities decide how intent is lowered.

thinking is the readable reasoning projection. blocks retains the exact provider continuation material when the provider requires it: Anthropic thinking blocks include their signature, and redacted_thinking blocks retain their opaque data. Treat those fields as private transcript data and do not render or edit them.

Replay is capability-owned rather than a caller option. The resolved route's reasoning_round_trip policy is strip by default, echo_signed for native Anthropic routes, and echo_same_key for routes such as Moonshot Kimi K3 that require reasoning under the same response field. This keeps private reasoning off unrelated providers while preserving valid continuation on routes that require it.

Tools#

Harn executes, approves, and audits tools. The provider executes provider_tools; Harn records their blocks but never dispatches them locally. Normally Harn steers a tool-bearing call away from a channel the capability catalog marks unsafe. A probe or matrix can force the requested channel with a non-empty tool_format_override_reason; Harn then preserves the requested format and, for native, sends the tool schemas even when the route does not advertise native tools. Provider-call transcript records expose the effective tool_format and native_tool_count, so the measured arm is visible at the wire boundary. Blank reasons do not bypass the catalog gates.

Cache, budget, and transport#

A streaming call emits nonempty replayed text as one delta, without another provider request. Cached predicate results follow the same rule. Recorded responses don't preserve token timing or chunk boundaries. Mock fixtures preserve their scripted chunk order.

speed: "fast" is admitted only when the selected route advertises a usable fast tier. Premium pricing applies only when provider telemetry confirms that tier. Explicit cache: true and prompt_cache_ttl require authored capability support because Harn must lower them to a provider-specific wire shape. A TTL must also appear in the route's prompt_cache_ttls list. cache: false is an opt-out, not capability intent. cache does not memoize full responses; use with_cache from std/llm/handlers for Harn-owned response caching.

Every provider_call_request transcript receipt records what happened to the prompt-cache breakpoint in prompt_cache_breakpoint.outcome:

A provider returns cache_read_input_tokens: 0 without an error both when no breakpoint was sent and when the cached prefix is below the model's minimum length. Read this field to tell the two apart.

Streaming applies three independent limits on every SSE and NDJSON provider: timeout_ms bounds the whole request, HARN_LLM_FIRST_TOKEN_TIMEOUT bounds the wait for the first chunk, and idle_timeout_ms (or HARN_LLM_IDLE_TIMEOUT) bounds later gaps. A provider must send its protocol's terminal event ([DONE], an OpenAI finish_reason, Anthropic message_stop, or Ollama done: true); EOF without one is a transient failure.

Caught stream failures carry source: "provider_stream", phase (awaiting_first_chunk or streaming), deadline (total, first_chunk, idle, or nil), and partial. The same fields appear on provider_call_error transcript receipts.

Schema-stream abort receipts carry a schema_failure record with kind, detail, path, and chunks_consumed. kind is one of invalid_json, invalid_schema, wrong_type, missing_required, unexpected_property, max_length, min_length, or constraint_violation. detail keeps the exact validator message for diagnosis. The error category remains schema_stream_aborted.

OpenAI Responses#

These keys require provider: "openai" and api_mode: "responses".

Provider options and observability#

Provider-specific fields are legal only below provider_options. They are never accepted as top-level provider names. Use this escape hatch for a native control that is not part of Harn's portable option contract; Harn does not reinterpret such fields as portable generation intent:

const result = harness.llm.call("hello", nil, {
  provider: "ollama",
  provider_options: {ollama: {num_ctx: 32768}},
})

Provider-native spellings of first-class generation controls are rejected in this escape hatch. For example, provider_options.ollama.repeat_penalty must become top-level repetition_penalty, and OpenAI top_logprobs must become logprobs: {top: ...}. This keeps capability admission, cache identity, routing, and replay on one typed path.

Model roles are ordinary defaults and compose with the same routing path:

const merged = harness.llm.call(prompt, nil, {
  model_role: "merge",
  output: schema,
})

Removed option names are not compatibility aliases. harn check and the runtime report the canonical replacement, so stale computed dicts cannot be silently projected away.

OpenAI Responses mode#

Set api_mode: "responses" with provider: "openai" when the call should use OpenAI's native Responses API instead of the generic /chat/completions adapter:

import { LlmCallOptions } from "std/llm/options"

const opts: LlmCallOptions = {
  provider: "openai",
  model: "gpt-5.4",
  api_mode: "responses",
  output: {schema: summary_schema, strict: true, validation: "error"},
  provider_tools: [
    {type: "web_search"},
    {
      type: "mcp",
      server_label: "docs",
      server_url: "https://mcp.example.com",
      require_approval: "always",
    },
  ],
  truncation: "auto",
  max_tool_calls: 4,
}
const result = harness.llm.call(
  "Search and summarize current docs.", nil, opts,
)

Use normal Harn tools when Harn should execute, approve, and audit a tool or MCP server locally. Use provider_tools only when the provider should execute a hosted tool or remote MCP connector. Provider-executed calls appear in result.blocks, transcript provider_payload.blocks, and provider_response_id metadata with executor: "provider_native" and the provider-native IDs. Set compact: true for a standalone Responses compaction pass; Harn records the opaque compaction items as private blocks so later turns can explicitly choose whether to feed the compacted provider window back as input.

Structural experiments can be enabled directly on a call:

import { LlmCallOptions } from "std/llm/options"

const experiment_opts: LlmCallOptions = {
  provider: "mock",
  structural_experiment: "prompt_order_permutation(seed: 42)",
}
const result = harness.llm.call(
  "Instruction\n\nContext block", nil, experiment_opts,
)

For custom transforms, pass a closure (or a std/experiments.custom(...) spec) that rewrites {messages, system} and returns either nil, a new message list, or {messages?, system?, metadata?}.

llm_call_structured#

harness.llm.call_structured(prompt, schema, options?) is the ergonomic helper for the "ask for JSON against this schema, retry on validation failure, return just the parsed data" pattern. It wraps harness.llm.call and pre-applies the schema-validated-JSON defaults so callsites stop repeating the same four options.

import { LlmCallOptions } from "std/llm/options"

const schema = {
  type: "object",
  required: ["name", "age"],
  properties: {
    name: {type: "string"},
    age: {type: "integer"},
  },
}
const structured_opts: LlmCallOptions = {
  provider: "anthropic", system: "You are precise.",
}
const person = harness.llm.call_structured(
  "Extract the speaker's name and age from the transcript.",
  schema,
  structured_opts,
)
harness.stdio.log(person.name)
harness.stdio.log(person.age)

Parameters#

Return value#

The validated data payload, typed as T when the schema is a Schema<T>. Throws on exhausted schema retries or transport failure — callers can assume the return matches the schema.

The {output: {schema, strict: true, validation: "error"}, schema_retries: 3} defaults are applied unless the caller overrides them in options.

Non-throwing variant#

harness.llm.call_structured_safe(prompt, schema, options?) returns the {ok, data, error} envelope (mirroring harness.llm.call_safe but with the validated .data pre-unwrapped) instead of throwing:

const r = harness.llm.call_structured_safe(
  prompt, schema, {provider: "openai"},
)
if !r.ok {
  harness.stdio.log(
    "structured call failed:", r.error.category, r.error.message,
  )
  return nil
}
const person = r.data

r.error.category is one of the canonical ErrorCategory strings ("rate_limit", "timeout", "schema_validation", "auth", "budget_exceeded", "transient_network", "generic", …) — match on the category instead of string-sniffing the message.

Diagnostic envelope variant#

harness.llm.call_structured_result(prompt, schema, options?) returns the full failure-mode envelope production agent pipelines need, so callers can keep raw model text, attempt counts, and validation / repair state without hand-rolling parse / repair chains. It never throws on transport or schema failures — ok: false plus error_category distinguishes the failure mode.

Set operation_timeout_ms to cap the entire structured operation, including schema retries, native-to-prompt fallback, and repair. This is distinct from timeout_ms, which bounds each provider request. On expiry the in-flight scope is cancelled and the envelope returns error_category: "timeout" plus the configured operation_timeout_ms.

import { LlmCallOptions } from "std/llm/options"

const result_opts: LlmCallOptions = {
  provider: "auto",
  schema_retries: 2,
  // Optional repair pass — runs only on malformed JSON or
  // schema-invalid output. Skipped on transport failures.
  repair: {
    enabled: true,
    model: "cheapest_over_quality(low)",
    max_tokens: 600,
  },
}
const r = harness.llm.call_structured_result(prompt, schema, result_opts)
if r.ok {
  const person = r.data
  // ...
} else {
  harness.stdio.log(
    "structured call failed:", r.error_category, "raw:", r.raw_text,
  )
}

Envelope fields:

Repair-pass semantics:

• After a schema or JSON miss, Harn first tries a local mechanical salvage: trailing commas, unquoted identifier keys, prose/fence extraction, and truncated closing braces. That tier spends no extra tokens. If the repaired text still fails the schema, it is rejected rather than coerced into a plausible shape.
• The repair block is recognized only by harness.llm.call_structured_result. Pass repair: {enabled: true, ...} to enable the LLM reissue; presence of the dict implies opt-in.
• The LLM tier runs at most once, with schema_retries: 0, only when local salvage did not produce valid JSON. It is skipped on transport failures because there is no raw text to salvage.
• Override keys (model, provider, max_tokens, system, …) are merged onto the main call's options for the LLM repair attempt.

Recover already-produced text#

harness.llm.recover_schema(text, schema, opts?) is the after-the-fact owner for already-produced text. Use it when you already have model text from harness.llm.call (or another producer) and want parse → extract → regex → optional LLM repair without issuing a new structured call first.

const raw = harness.llm.call("Summarize the ticket").text
const r = harness.llm.recover_schema(raw, schema)
if r.ok {
  harness.stdio.log(r.data)
} else {
  harness.stdio.log("recovery failed:", r.stage, r.error_category)
}

Set {repair: false} for a fully deterministic pass. The optional LLM repair block accepts the same overrides as harness.llm.call_structured_result. See the language spec for the {ok, data, raw_text, error, error_category, attempts, stage, repaired} envelope and the parsed / extracted / regex / llm_repair stages.

When to use which helper#

• Product code that needs just the parsed payload: prefer harness.llm.call_structured. It removes the output, schema_retries, and .data boilerplate from every callsite.
• Code that also needs token counts, transcript, thinking traces, or to pass a pre-built transcript: call harness.llm.call directly and read .text / .data / .usage.input_tokens / etc. off the full result dict.
• Call sites that prefer explicit branching over try blocks: harness.llm.call_structured_safe (the non-throwing envelope).
• Production agent pipelines that need raw-text retention, attempt counts, and an optional repair pass on malformed JSON: harness.llm.call_structured_result — replaces the harness.llm.call → response.data → try { json_parse(...) } → json_extract → repair → schema_check chain that downstream callers would otherwise hand-roll.

Composable callers#

agent_loop accepts an llm_caller: option — a closure that owns each turn's harness.llm.call(...). Wrap it with middleware from std/llm/handlers (retry / fallback / shadow / logging / budget / cache / circuit breaker) to compose resilience without forking the loop:

import {default_llm_caller} from "std/llm/caller"
import {with_retry} from "std/llm/handlers"

const caller = with_retry(default_llm_caller(), {max_attempts: 4})

const result = agent_loop(harness, task, system, {
  loop_until_done: true,
  llm_caller: caller,
})

llm_caller(opts = nil) is the blessed default stack — it is exactly with_retry(default_llm_caller(), opts?.retry ?? {}) with typed reserved-status classification and billed-empty re-dispatch on by default. Reach for it instead of re-composing retry by hand, and compose with_cache / with_budget / with_logging around it when you need more:

import {llm_caller} from "std/llm/caller"

const result = agent_loop(harness, task, system, {
  loop_until_done: true,
  llm_caller: llm_caller({retry: {max_attempts: 4}}),
})

Caller contract:

fn(call) -> {ok: true, value: <llm dict>}
          | {ok: false, status: <reserved>, error?: any, retryable?: bool}
//   call = {prompt, system, opts, turn: {iteration, session_id, attempt}}

with_retry's max_attempts: N counts total attempts. Migrating llm_retries: K (removed in 0.10): pass max_attempts: K + 1 — the removed option counted retries after the first attempt.

See Composable callers and middleware for the full module catalog (handlers, ensemble, refine, budget, defaults, safe, prompts, catalog).

llm_completion#

Use harness.llm.completion for text continuation and fill-in-the-middle generation. It lives at the same abstraction level as harness.llm.call.

import { LlmCallOptions } from "std/llm/options"

const completion_opts: LlmCallOptions = {
  provider: "ollama",
  model_tier: "small",
}
const result = harness.llm.completion(
  "const total = ", ";", nil, completion_opts,
)
harness.stdio.log(result.text)

Cost tracking#

Harn provides builtins for estimating and controlling LLM costs:

// Estimate cost for a specific call
const cost = llm_cost("claude-sonnet-5", 1000, 500)
harness.stdio.log("Estimated cost: $${cost}")

// Check cumulative session costs
const session = harness.llm.session_cost()
harness.stdio.log("Measured cost (nil if unknown): ${session.total_cost}")
harness.stdio.log("Budget charged: $${session.budget_charged_usd}")
harness.stdio.log("Calls: ${session.call_count}")
harness.stdio.log("Input tokens: ${session.input_tokens}")
harness.stdio.log("Output tokens: ${session.output_tokens}")

// Set a budget (LLM calls throw if exceeded)
harness.llm.budget(1.00)
harness.stdio.log("Remaining: $${harness.llm.budget_remaining()}")

For per-call controls, pass a budget envelope on harness.llm.call (the typed shape is LlmBudget from std/llm/options):

import { LlmBudget, LlmCallOptions } from "std/llm/options"

const budget: LlmBudget = {
  max_cost_usd: 0.001,
  max_input_tokens: 8000,
  max_output_tokens: 1024,
}
const budgeted_opts: LlmCallOptions = {
  provider: "openai",
  model: "gpt-5.4-mini",
  max_tokens: 1024,
  budget: budget,
}
const result = try {
  harness.llm.call("Summarize this", nil, budgeted_opts)
}

Harn estimates prompt tokens before the provider request leaves the process, using tiktoken-rs for known OpenAI models, labeled tiktoken approximations for Claude/Gemini families, and a heuristic fallback for unknown model IDs. It then projects cost with the provider/model pricing table and throws a terminal budget_exceeded dict when a limit would be exceeded. In a try { ... } expression that surfaces as Result.Err({kind: "terminal", reason: "budget_exceeded", projected_cost_usd: ...}).

agent_loop accepts the same envelope. max_* limits apply to each model turn; total_budget_usd is an aggregate loop budget and exits gracefully with status: "budget_exhausted" before starting a turn that would exceed it.

Conservative admission#

The default budget uses request estimates and observed output/cache usage. For before-transport monetary admission, opt in on the first call of an execution:

import { LlmBudget } from "std/llm/options"

const bounded: LlmBudget = {
  admission: "conservative", total_budget_usd: 0.60,
}
const answer = try {
  harness.llm.call("Return one word", nil, {
    provider: "openai", model: "gpt-6-luna",
    max_tokens: 64, budget: bounded,
  })
}
const admission = harness.llm.session_cost().admission

This mode reserves the full catalog context at the highest applicable input and cache rates, plus the requested output limit at the highest output rate. It does not use a tokenizer estimate, past short completions, or cache hits to reduce the next reservation. A known-price attempt that cannot fit is refused before provider transport. An explicit max_cost_usd also limits each attempt.

One execution and its inherited inline/spawned work share the reservation ledger. The first conservative call fixes the ceiling; later calls may tighten it but cannot widen it or disable it by omitting budget. Activation after an unreserved attempt is refused. Independent top-level executions have separate ledgers. Allocate disjoint budgets to separate processes: for an experiment, actor and grader allocations across both arms must sum to the whole allowance. The runtime does not turn separate process budgets into a shared batch cap. For a runnable actor/grader sequence with disjoint process allocations, see Run an experiment inside one allowance.

Full, streaming, offthread, retry, and fallback attempts use the same admission boundary. Complete, consistent provider token counters release only the unused part of the reservation, at conservative rates without cache discounts. Errors, cancellation, and missing usage retain the full amount as uncertain, including failures that might have occurred before sending. They are never free retries.

Supported billing shapes are direct OpenAI text and direct Anthropic text with ordinary five-minute caching, exact catalog pricing, a context limit, and a positive output cap. Complete Anthropic fresh-input, cache-read, cache-write, and output counters settle at conservative rates. Missing cache categories retain the full input-context bound and release only unused output allowance; missing categories are never treated as known zero. This can exhaust an allowance well before actual spend; read the separate admission receipt. One-hour caching is refused even when requested in an inline cache-control block. Premium serving, media, hosted provider tools, opaque provider overrides, expiring promotional rates, and unpriced or other provider routes are refused. Fill-in-the-middle completion, provider conformance probes, and healthcheck/warm-up operations also refuse an active conservative scope; run any required preflight separately under its own allocation before starting the bounded execution. Generic HTTP, connector and subprocess charges are outside LlmBudget. Ordinary function tools are supported. This is enforcement against the catalog's provider contract, not an invoice guarantee: if reported usage exceeds the reserved bound, Harn records it, reports a contract violation, and refuses subsequent attempts.

harness.llm.session_cost().admission is absent in default mode. When active it contains mode, ceiling_usd, settled_upper_usd, in_flight_usd, uncertain_usd, denied_attempts, and contract_broken. Monetary fields are exact decimal strings. A known output-limit or route violation stops further admission even when missing usage prevents complete settlement. Denials retain the terminal budget_exceeded contract and add a typed admission_reason, such as unknown_pricing, unsupported_billing_shape, or insufficient_allowance. harness.llm.call_safe(...).error declares both admission_reason and provider_call_count as optional fields, so scripts can read the reason and measured dispatch count directly.

These are reservation facts, separate from the existing actual-usage totals and their unknown-usage indicators; settled_upper_usd is not billed spend.

Native host budgets#

Rust embedders can retain one harn_vm::llm::ConservativeLlmBudget handle and wrap their futures with scope. Its receipt exposes the same reservation facts. Cloning a handle shares its ledger. Module initialization and later VM entries within that scope consume one allowance; asynchronous suspension does not expose it to another host task. A nested handle may tighten the parent but cannot replace it with a different allowance.

Served calls opt in with @budget(llm_admission: "conservative", llm_cost_usd: 0.6). The scope starts before loading module code and covers the handler. Each independent served call gets a separate allowance, so callers must allocate across calls themselves.

An ACP embedder sets BudgetSpec.llm_admission to Some(AdmissionMode::Conservative) alongside llm_cost_usd. The live session retains its ledger across prompt turns and shares it with session forks. Changing the session budget cannot raise or remove an activated ceiling. Configure this before the first prompt: cold-restored sessions cannot prove their prior reservation state and refuse conservative admission. Start a new independently allocated run instead. Durable worker declarations currently refuse this mode before module execution; their persisted grant lifecycle is not yet supported. These host scopes still do not allocate across processes unless the host installs a machine spend quota.

Machine spend quota for native hosts#

harn_vm::MachineSpendQuota provides a durable UTC daily and calendar-month ceiling across processes. The host opens the same private SQLite path and billing-scope ID for every session belonging to one person. Policy amounts are integer micro-USD, so 1_000_000 is one dollar. A host wraps the whole model execution tree in quota.scope(future) before its first provider attempt. A ConservativeLlmBudget created within that scope may impose a tighter per-session ceiling. Retries, fallbacks, streaming, and spawned calls use the same pre-transport reservation boundary. Unsupported billing shapes and unknown prices fail closed. The provider registry's self-hosted runtimes are known-zero and consume no monetary allowance.

The quota reserves the catalog upper bound atomically before transport. A process crash, cancellation, or missing usage leaves that bound reserved; only complete provider usage can release a proven unused portion. The receipt keeps priced observed usage and the count of attempts without confirmed usage separate from the reserved allowance; neither is a provider invoice. An observed provider contract violation latches a durable refusal, including after restart, so later calls cannot rely on an invalid cost bound. A quota exhaustion error reports the limiting day or month, remaining allowance, and the UTC reset instant. Sessions in separate projects must use the same path and billing-scope ID to share one ceiling. While the scope is active, harness.llm.session_cost().machine_spend carries the same typed receipt; it is absent when no machine quota is installed. harness.llm.budget_remaining() reports the tightest remaining session, execution, or machine allowance that is set.

The first opener records the policy. A later opener with a different policy is refused. An authorized host administrator can call update_policy with a nonempty approval reference; the change and previous limits are recorded in the same database transaction. session/set_budget remains a per-session control and cannot silently raise the durable machine ceiling. The host owns authentication of the administrator and the database path; scripts running in the VM have no direct policy-update operation. Reservations protect cataloged provider charges, not external connector, tool, or platform fees.

Import std/llm/budget for reusable helpers such as estimate_text_tokens_detail(text, model), which includes the encoder label (cl100k_base, o200k_base, etc.) and whether the count is exact or an approximation.

Testing with mock LLM responses#

The mock provider returns deterministic responses without API keys. Use harness.llm.mock_enqueue() to queue specific responses — text, tool calls, or both:

// Queue a text response (consumed in FIFO order)
harness.llm.mock_enqueue({text: "The capital of France is Paris."})
const r = harness.llm.call(
  "What is the capital of France?", nil, {provider: "mock"},
)
assert_eq(r.text, "The capital of France is Paris.")

// Queue a response with tool calls
harness.llm.mock_enqueue({
  text: "Let me read that file.",
  tool_calls: [{name: "read_file", arguments: {path: "src/main.rs"}}],
})

// Queue token logprobs for confidence/reranking tests
harness.llm.mock_enqueue({
  text: "certain", logprobs: [{token: "certain", logprob: 0.0}],
})

// Pattern-matched mocks (reusable by
// default, matched in declaration order)
harness.llm.mock_enqueue({text: "I don't know.", match: "*unknown*"})
harness.llm.mock_enqueue({
  text: "step 1", match: "*planner*", consume_match: true,
})
harness.llm.mock_enqueue({
  text: "step 2", match: "*planner*", consume_match: true,
})

// Provider-style error envelopes exercise the same catch/safe-call paths
// as live provider failures.
harness.llm.mock_enqueue({
  error: {status: 503, kind: "transient", reason: "upstream_unavailable"},
})

// Inspect what was sent to the mock provider
const calls = harness.llm.mock_calls()
// Each entry includes mock_scope,
// messages, system, tools, output/thinking
// controls, and portable generation options
// such as temperature and max_tokens.

// Clear all mocks and call log between tests
harness.llm.mock_clear()

For concurrent agent work, prefer with_llm_script(harness.llm, responses, fn) from std/testing. It scopes fixture installation and cleanup to the supplied HarnessLlm; helpers that only receive another capability cannot inspect or mutate the queue. Use harness.llm.mock_snapshot() and harness.llm.mock_calls() for queue and call evidence. The JSONL fixture parser is a runtime-internal implementation seam, not a script interface.

When no harness.llm.mock_enqueue() responses are queued, the mock provider falls back to its default deterministic behavior (echoing prompt metadata). This means existing tests using provider: "mock" without harness.llm.mock_enqueue() continue to work unchanged.