Provider catalog refresh workflow

scripts/update_provider_catalog.harn is the Harn-native workflow that periodically collects model availability, pricing, and capability signals from provider sources, normalizes them, and emits:

• a markdown drift report under .harn-runs/provider_catalog/drift-report.md;
• a TOML candidate patch under .harn-runs/provider_catalog/candidate.toml;
• a JSON evidence manifest under .harn-runs/provider_catalog/refresh.json, with adapter coverage, observations, provenance, conflicts, and typed drift.

Refresh compares the stored rate card, including promotions, schedules, and input-token bands. harness.llm.provider_catalog() exposes that complete card without resolving it at the current time. Cost estimates and completed-call receipts resolve prices through the pricing engine, so an off-peak refresh does not mistake a scheduled discount for a changed base rate.

Price comparison ignores source links, review notes, and generated window or promotion names. It expands weekday groups and applies the pricing engine's default cache multipliers before comparing schedules. Rates, dates, time boundaries, weekdays, UTC offsets, token thresholds, and tool fees still count as price changes. Reports and candidate patches retain the original records.

The workflow never mutates the shipped catalog. The patch is a review aid: diff it against the TOML fragments under crates/harn-vm/src/llm/catalog_sources/, the capability fragments under crates/harn-vm/src/llm/capability_sources/, or your project's harn.toml overlay before landing changes.

Keep the provider's durable rate card in the main pricing fields. Put a dated discount in pricing.promotions. Harn uses an active promotion for cost estimates and returns to the durable rate when it ends. Use review_after when a provider gives a minimum duration but no firm end date.

Recurring prices belong in pricing.schedules: named weekday windows with a fixed UTC offset, inclusive start, exclusive end, and rate multipliers. A window may cross midnight. Overlaps fail catalog validation. OpenRouter's pricing.overrides records are normalized to a peak base card and discount windows, including all-day records with omitted time bounds. DeepSeek's Chinese-public-holiday exemption remains an explicit approximation in the curated row; Harn does not maintain a holiday calendar.

Completed calls resolve the card at request start, then apply the input band, cache lifetime, and serving tier. Usage receipts record the selected card and cache_ttl_unpriced when the route has no one-hour write price. Hosted tool counts and audio-token counts use hosted_tool_fees and modality_rates. Unpriced reported units keep the known token cost visible but leave the budget projection unbounded. Monthly free allowances are account state and remain unapplied, named on the receipt.

platform_fee_percent allocates funding overhead when catalog prices estimate a request's cost. OpenRouter's 5.5 percent default assumes standard card-funded credits and excludes the minimum top-up fee. The receipt calls this platform_fee_estimate_usd with basis catalog_estimate_funding_route_unknown; it is not an invoice line, does not describe BYOK or alternate funding, and is never added to a provider-reported authoritative total. Override the provider field to zero when that funding assumption does not apply.

The markdown report includes aggregator discoveries for awareness, but the candidate TOML contains only provider-owned, high-confidence changes and additions with actionable pricing or capability metadata. This keeps broad discovery APIs such as OpenRouter's useful without turning bare model IDs or uncurated mirrors into apparent review-ready catalog rows. Catalog identity is always the (provider, model id) pair; equal model ids on different providers are compared independently.

Running the workflow#

# Default: replay against bundled fixtures, write report + candidate
# under .harn-runs/provider_catalog/.
harn provider catalog refresh

# Live mode: hit real provider sources. Key-required adapters attach
# the configured auth header from env vars; missing keys produce a
# "skipped" adapter record. --json prints a compact coverage summary;
# the full machine-readable evidence is in refresh.json.
harn provider catalog refresh --live --json

# CI gate: same fixture replay, but compare against the committed
# goldens at scripts/provider_catalog_fixtures/expected_*.
harn provider catalog refresh --check

# Refresh the committed goldens after intentional adapter changes.
harn provider catalog refresh --check --update

The JSON summary distinguishes complete, partial, and unmeasured runs and reports adapter, empty-source, and observed-model counts. A live run with no observed models fails even when every adapter returned a nominal response. Chat-model indexes do not count decision-only routes as removals.

The command stops a refresh that exceeds 120 seconds and reports a timeout as a failed, incomplete refresh rather than an empty catalog. Pass --timeout-secs <seconds> to choose a different bound, or --timeout-secs 0 to wait indefinitely. Fixture mode is the default and does not grant the child process network access; --live selects live sources and adds that grant explicitly.

The CI gate is wired into make check-provider-catalog-drift and runs from make all.

Trusted change notices#

scripts/provider_catalog_notice.harn handles narrow provider change notices between broad refresh runs. It consumes a transport-neutral JSON record:

{
  "source_id": "provider-pricing-2026-08",
  "retrieved_at": "2026-07-28T12:00:00Z",
  "provider": "anthropic",
  "text": "The original notice text, if available.",
  "source_url": "https://provider.example/immutable-notice",
  "provenance": "verified_link"
}

An email label, webhook, or document-store adapter only needs to write this shape. provenance is authenticated, verified_link, unverified, or failed; the latter two fail closed. At least one of text or an immutable source_url is required.

The model is limited to a schema-constrained extraction:

• addition, price, promotion, retirement, endpoint, and capability are distinct typed variants;
• provider, model id, and supporting evidence are retained, with effective dates and old/new values for changes that require them;
• the model never receives a file mutation tool and never emits TOML.

For an existing model, extraction sees that provider's catalog IDs, names, and current prices. An uncertain identity is rejected for review; an unknown ID in a price or retirement candidate cannot masquerade as a new-model addition.

Deterministic code resolves the provider and model identity. For an existing model, it verifies the old value, finds exactly one owning source table, and applies the constrained edit. Duplicate identities are rejected. An unsupported provider records a no-op; a new model on a supported provider produces an incomplete proposal listing context, pricing, capability, and routing verification still required.

An addition needs no invented launch date and cannot insert a catalog row without the listed verification. The receipt retains the structured extraction that led to the disposition.

The extractor emits at most one candidate per run. Split a notice with several changes into focused records and account for every remaining fact.

Capability edits target the generated capability matrix's owning fragments, not legacy model tags. They require one exact model_match rule; a wildcard-derived family rule is deliberately rejected as ambiguous rather than silently changing sibling models.

Use a checked extraction fixture to inspect the plan without an LLM call:

harn run scripts/provider_catalog_notice.harn -- \
  --notice scripts/provider_catalog_notice_fixtures/carried-update.json \
  --extraction scripts/provider_catalog_notice_fixtures/carried-update.json

When either input is outside the worktree, grant its containing directory before the script path. Keep the run sandboxed and repeat --read-only-root when the notice and checked extraction come from different directories:

notice_path=/path/to/notice.json
extraction_path=/path/to/extraction.json
harn run \
  --read-only-root "$(dirname "$notice_path")" \
  --read-only-root "$(dirname "$extraction_path")" \
  scripts/provider_catalog_notice.harn -- \
  --notice "$notice_path" \
  --extraction "$extraction_path"

For a live extraction, omit --extraction and optionally choose --provider and --model. Without --model, Harn uses that provider's catalog default; it does not pair a cross-provider tier alias with the selected provider. The live extraction schema requires a candidate field, allowing null when the notice has no catalog change. This keeps providers from omitting the change record while returning classification candidate. --max-cost-usd caps its one model call at $0.10 by default; --extraction-timeout-secs caps that call at 90 seconds by default. By default the script only writes an idempotent local receipt below .harn/provider-catalog-notices/. An incomplete new model writes a proposal beside the receipt, even with --apply --open-pr, and leaves Git untouched. A validated patch with --apply requires a clean, dedicated worktree and regenerates the catalog, matrix, and support projections before their checks. Pass --harn-bin /absolute/path/to/harn to use one already built Harn binary for generation and checks. The workflow runs the catalog's deterministic refresh, artifact, matrix, and support gates. CI also runs the sandbox and documentation snippet gates outside this nested Harn execution. --open-pr signs the commit, stages only tracked changes, and opens a draft PR on a stable branch derived from the notice digest. It never enables merge or auto-merge. A repeated notice returns the existing PR.

git worktree add ../harn-provider-notice origin/main
(
  cd ../harn-provider-notice
  make setup
  notice_path=/path/to/notice.json
  harn run \
    --read-only-root "$(dirname "$notice_path")" \
    scripts/provider_catalog_notice.harn -- \
    --notice "$notice_path" \
    --repo-root "$PWD" \
    --apply \
    --open-pr
)

Receipts stay below the dedicated worktree by default. If an adapter instead passes an absolute external --output-dir, create that directory first and grant that exact directory with --write-root before the script path. A read-only root does not authorize receipt writes.

The optional cron package in examples/triggers/provider-catalog-notice/ shows how a public adapter can place a neutral notice file and invoke this workflow. The scheduler needs only the worktree and draft-PR authority granted to that process; private mailbox or product state is not part of the workflow contract.

Measuring tool-call routes#

Use the built-in probe for one route. It sends the request through the same provider adapter, request builder, stream parser, and response normalizer as harness.llm.call. No probe script or temporary Harn file is needed.

# Inspect the request without calling the provider.
harn provider tool-probe openai --model gpt-6-sol --dry-run-request

# Check non-streaming and streaming native tool calls.
harn provider tool-probe openai --model gpt-6-sol --mode both

The probe reads the provider's normal API-key environment variable. Pass --base-url only to test a raw compatible endpoint; that override bypasses the provider adapter. Each JSON result records the observed tool-call shape, latency, token use, and catalog-priced cost.

Use the spend-capped campaign to measure several routes or repeat a stochastic claim. Dry-run mode performs request and credential readiness checks without provider calls:

harn run --no-sandbox scripts/provider_tool_probe_campaign.harn -- \
  --route openai:gpt-4.1-nano \
  --behavior provider_tool_probe \
  --repeat 3 \
  --max-cost-usd 5 \
  --output-dir .harn-runs/provider-tool-probe-campaign/openai-nano

Add --live only after reviewing that plan. Live repeats are executed as separate probe processes so the campaign accounts for every physical request before starting the next one. The campaign stops when the observed runtime-priced cost reaches --max-cost-usd; it also fails closed after an unpriced case instead of continuing with an unknown ledger. --max-probes limits selected route/behavior/mode units independently of the dollar cap.

Each live output directory contains:

• report.json and report.md, including parseability, empty completions, latency, request/retry incidence, rate limits, token usage, and spend;
• the individual tool-probe-*.json provider receipts;
• scorecard-inputs.json, a manifest binding every input hash to the Harn source revision, exact campaign-source hash, and provider-catalog hash;
• deterministic scorecard.json and scorecard.md artifacts generated by harn provider tool-scorecard;
• scorecard-receipt.json, which hashes the inputs and both projections.

The tool probe deliberately performs no automatic retries, so retry_count is zero and request_attempt_count is the physical request count. A 429 is recorded separately as a rate-limited case. Suggested catalog changes in the scorecard remain review-only data patches; the campaign never edits catalog TOML.

Measuring portable option claims#

An option can depend on an explicitly selected effort. A capability rule with portable_option_effort_support = { temperature = ["none"] } admits temperature only when the resolved request selects effort: "none" and the route supports that effort. This condition overrides the route-wide temperature flag for that request. An omitted effort does not satisfy it, even when Harn represents both omission and explicit none as disabled reasoning. Direct GPT-6 Sol and Luna use this condition. OpenRouter clears the grant because its endpoints do not advertise temperature, even with none; structured evaluations there omit sampling rather than relying on a tolerated, unforwarded field.

Use harn provider option-probe when the catalog says an endpoint accepts or rejects one of Harn's portable generation options. The command makes one small request through the normal provider adapter and compares the provider's answer with the exact catalog field:

# No provider call. Shows the endpoint, claim, field, and request count.
harn provider option-probe anthropic \
  --model claude-opus-4-7 \
  --option temperature \
  --plan --json

# One live request. Exit 1 means drift; exit 2 means no provider measurement.
harn provider option-probe anthropic \
  --model claude-opus-4-7 \
  --option temperature \
  --fail-on-drift --json

The typed option vocabulary is temperature, top_p, top_k, seed, frequency_penalty, presence_penalty, and stop. Each report names the provider and model endpoint, catalog field, declared value, request count, measured count, provider verdict, and one of match, drift, or unmeasured. Authentication, throttling, missing models, and local admission failures are unmeasured; they cannot turn an absent provider observation into a match. So are unrelated terminal outcomes such as content-policy blocks. Only the provider error taxonomy's structured invalid_request reason proves that the wire shape was rejected. The attempt owns its physical request count: a local gate reports zero requests and remains unmeasured, while a provider response or served-empty response reports one.

Every rejection needs a successful control request without the selected option. If that control also fails, the option remains unmeasured with failure_class: "control_failed". A timeout gets one immediate retry; a rate limit gets one retry after the runtime's retry_after_ms, or one second when it is absent. The HTTP error owner caps that hint at 60 seconds and preserves whether the provider requested a longer deadline. An over-cap provider deadline or typed delay remains unmeasured without a retry. A persistent rate limit remains unmeasured. Receipts include the retry and control in request, usage, and cost counts; an unpriced request remains an accounting gap even when a later request has a price.

Ungated OpenRouter probes require endpoints that advertise all requested parameters. An explicit parameter-routing refusal is classified as invalid_request; a failure involving another routing filter remains unmeasured. Acceptance proves advertised support and request acceptance, not semantic enforcement of the option.

An OpenRouter account privacy refusal produces excluded_by_account_policy with the provider message and no observed option-support value. The campaign counts these exclusions separately from eligible unmeasured cells. An exclusion cannot satisfy either measured control direction, and a campaign containing only exclusions remains incomplete. Preserve the account privacy settings.

Normal calls use the catalog to reject or remove unsupported options before egress. A truthful negative probe must let its selected option reach the provider, or it can only confirm the catalog against itself. The CLI selects probe authority in a typed async-task scope, then option extraction captures it in the resolved call contract and carries it through spawned transport work. It suspends shaping for the selected option only; sibling calls and unrelated options keep normal catalog policy. Pass --gated for a confirm-only run that leaves every guard enabled.

An accepted verdict proves that the endpoint accepted a meaningful, non-default value on the wire. It does not prove that a provider honored the value semantically; providers that silently ignore unknown fields require a separate output-sensitive conformance test.

This distinction matters because provider APIs change at model boundaries. Anthropic's current Messages API reference says models after Claude Opus 4.6 reject non-default temperature, top_p, and top_k. Google's model resource exposes whether a Gemini model uses top-k sampling; an empty topK means requests may not set it. The catalog records those endpoint-specific facts instead of treating a field present in an SDK type as universally supported.

The existing spend-capped campaign executes every portable option without a second orchestration path:

harn run --no-sandbox scripts/provider_tool_probe_campaign.harn -- \
  --catalog-routes \
  --exclude-local \
  --behavior provider_option_probe \
  --max-probes 20 \
  --max-cost-usd 2 \
  --output-dir .harn-runs/provider-option-probes

Review the dry run, then add --live. A live option campaign is incomplete unless it contains at least one measured-supported and one measured-unsupported control, with no unavailable or eligible unmeasured option cells. Its claim_controls receipt counts support, non-support, account-policy exclusions, unavailable backends, and unmeasured cells separately. A changing pool's controlled rejection measures current compatible-backend availability rather than stable non-support; its probe and control receipts retain the selected backend identities. Missing selection remains unmeasured. Individual option-probe-*.json receipts join the same catalog hash, runtime fingerprint, sharding, credential-name readiness, and budget ledger as tool-call receipts. Credential-missing option cells count as unmeasured in a live campaign; account-policy exclusions have their own count. Partial eligible endpoint coverage cannot satisfy the control. Pass --provider to declare the providers a run is responsible for; a declared provider with no credential still fails the run. The weekly Provider contract probe workflow declares its providers in PROBE_PROVIDERS next to the secrets it receives. Catalog routes marked availability = "dedicated" are not selected, because a serverless request to a model the provider serves only from a deployment returns not-found whatever the option.

Fixture mode intentionally covers only deterministic public sources. Live mode adds provider-owned /models sources across the hosted provider set: Anthropic, OpenAI, Hugging Face Router, Gemini, Mistral, Cohere, xAI, Together, Groq, Cerebras, DeepSeek, Fireworks, DashScope, MiniMax, Z.AI, Moonshot, Baseten, DeepInfra, SambaNova, NVIDIA NIM, Nebius Token Factory, FlexAI, Hyperbolic, SiliconFlow, Parasail, and Atlas Cloud. For NVIDIA, set NVIDIA_API_KEY; set NVIDIA_NIM_BASE_URL only when you need a self-hosted NIM or gateway URL. The built-in NVIDIA default is https://integrate.api.nvidia.com/v1.

Reasoning-effort ladders#

reasoning_effort_levels is the one capability row nothing else verifies. Tool formats have provider tool-probe, dispatch has provider dispatch-audit, pricing has the rate card — but effort ladders were written by hand into capability fragments and re-asserted by hand in unit tests, so a wrong rung stayed invisible until a caller took a provider error in production. Harn shipped glm-5.3 declaring a medium rung that Z.AI rejects with HTTP 400 until a probe caught it.

harn provider effort-probe closes that loop by sending one small live call per rung and reporting both drift directions:

harn provider effort-probe --model glm-5.3 --suggest-fragment
harn provider effort-probe --all-declared --one-per-claim --plan
harn provider effort-probe --all-declared --one-per-claim --fail-on-drift --json

Every probed rung is a paid request and --all-declared selects every route that declares a ladder, so --plan lists the routes and the resulting call count without calling anything.

Catalog ladders are rules with a model_match pattern, so one claim can back dozens of routes and probing all of them re-measures the same rule and bills for it each time. --one-per-claim probes one representative per distinct (provider, ladder) claim, which covers the whole claim space for a fraction of the calls. Each result carries covers, the routes its verdict stands for, so a narrowed run never reads as though it had probed everything.

• declared_but_rejected — the catalog promises a rung the route refuses. Callers asking for it take a provider error. This is the dangerous direction.
• accepted_but_undeclared — the route serves a rung the catalog omits. Harn's ladder snap silently redirects callers away from a working rung, so nothing breaks but the route is quietly less capable than it is.

The probe stands down every catalog-derived rewrite of the requested effort, not just the declared-ladder check. Each dialect used to grow its own || probe_ungated hedge around one omit site; leaving any one of them on lets the catalog confirm itself. The dropped-parameter case is the worst: the request succeeds without the rung reaching the wire and the probe records an acceptance the route never gave. That is how none on OpenRouter GLM-5.3 came back accepted — the top-level field was ungated, the nested enabled: false skip was not. One policy (catalog_may_shape_requested_reasoning) is consulted at every omit across OpenAI-compatible, Gemini, and Anthropic, so that class of lie is unrepresentable. Pass --gated to keep the checks on for a confirm-only run; rungs they block are reported as gated_locally rather than counted as provider verdicts, so a gated run never invents drift it did not observe.

accepted means the route served the request Harn built for that rung, not that it honoured the rung. Three limits follow, and the probe is explicit about each rather than hiding them:

• An OpenAI-compatible endpoint that ignores unknown parameters answers 200 for every rung it has never heard of. Each accepted attempt records output_tokens and reasoning_tokens when the provider reports them; usage that does not move across three or more accepted rungs is reported on the route as usage_unmoved rather than left for a human to notice. That is a warning, not drift: a swallowed parameter is not a catalog contradiction.
• On a provider where Harn translates effort instead of sending it verbatim, acceptance describes Harn's projection rather than a provider ladder. Gemini maps effort to a thinking budget and resolves high, xhigh, and max to the same budget, so all three are served and none of them distinguishes a provider-side level. Runs that touch such a route print a note saying so.
• An empty reasoning_effort_levels means "unknown/all", not "nothing allowed". Harn snaps no caller and refuses no rung for such a route, so an accepted rung contradicts nothing and is reported as a measurement rather than as drift.

A served-but-empty completion counts as accepted. A reasoning route spends the response cap on reasoning first, so a cap tight enough to leave no visible content makes every accepted rung come back as empty_generation — a fact about --max-tokens, not about whether the rung is valid. Reading it as a refusal produced ladders with holes in the middle that no provider implements.

--suggest-fragment prints a corrected reasoning_effort_levels row for each drifting route. Like the tool-probe campaign, it is review-only: the probe never edits catalog TOML.

Effort ladders are route-specific, not model-specific. The same weights served through two providers can expose different rungs: Z.AI's glm-5.3 accepts low/high/max, while the OpenRouter mirror of the same model accepts minimal through max because OpenRouter normalizes effort itself. Both refuse none — GLM-5.3 always reasons. Probe each route you catalog rather than copying a ladder across a mirror.

One claim, not three#

A fragment can say "this route takes an effort ladder" three ways: thinking_modes containing effort, the legacy reasoning_effort_supported flag, and a non-empty reasoning_effort_levels. thinking_modes owns the answer, and the other two are absorbed into it — each can only ADD effort, never remove it. Setting reasoning_effort_supported = false is therefore not a way to remove an effort mode the row declares; leave effort out of thinking_modes and make no claim elsewhere.

Absorbing rather than comparing makes the contradiction unrepresentable instead of merely detectable, which is why there is no separate lint for it. Nineteen shipped routes had been contradicting themselves, and the failure was invisible because both halves of Harn read different fields: the reasoning policy read the flag and produced an effort request, and the option validator read the modes and refused the request the policy had just built — with the same message a route that genuinely has no effort control produces.

The one residual invariant, that a declared ladder implies the effort mode, is asserted for every catalogued route by effort_capability_agrees_across_every_catalogued_route.

Prefer the canonical spelling in new fragments:

thinking_modes = ["enabled", "effort"]
reasoning_effort_levels = ["low", "high", "max"]  # measured, not asserted

Catalog source and generated artifacts#

Harn authors edit small TOML fragments under crates/harn-vm/src/llm/catalog_sources/. Harn generates the embedded runtime snapshot at crates/harn-provider-catalog/src/providers.toml from those fragments. Provider capability rules use the same pattern: crates/harn-vm/src/llm/capability_sources/ generates crates/harn-provider-catalog/src/capabilities.toml.

00-base.toml names the default_provider, used when a call names neither a provider nor a model. 40-defaults/provider-models.toml records each provider's runtime and portal default; they may differ deliberately. A provider without a runtime entry has no default model, and a call naming only that provider fails before dispatch rather than borrowing another provider's model. The retired top-level fallback_model key is still accepted in overlays, and becomes the runtime default of the overlay's default provider. Keep those routes in the catalog rather than in Rust match tables. Catalog generation rejects unknown, provider-mismatched, or deprecated choices there, in qc_defaults, the three tier aliases, and model ladders. Host environment overrides for local models remain runtime input, so the built-in catalog cannot validate their IDs.

# Regenerate providers.toml, capabilities.toml, and all checked-in
# provider catalog artifacts from source fragments in one hermetic pass.
harn provider catalog generate

# CI gate: fail if any generated provider catalog artifact drifted.
harn provider catalog generate --check

Direct edits to crates/harn-provider-catalog/src/providers.toml and crates/harn-provider-catalog/src/capabilities.toml are invalid. The files are checked in so Harn can still embed known-good offline snapshots with include_str!, but make check-provider-catalog proves every checked-in provider catalog projection matches the fragments.

One capability rule may cover exact aliases without copying its body:

[[provider.gemini]]
model_match = ["gemini-3.7*", "models/gemini-3.7*"]
native_tools = true

The list is one ordered rule. Keep separate rows when precedence or capability values differ. Model routes stay separate across providers because their prices, wire IDs, quotas, and availability can change independently.

Harn also checks in generated artifacts under spec/provider-catalog/ so downstream hosts do not need to parse Harn internals:

• provider-catalog.json — normalized providers, models, aliases, variants, QC defaults, capabilities, pricing, family/lineage metadata, reviewer-diversity hints, deprecation metadata (including the structured superseded_by migration pointer), fast-mode tier metadata (the accelerated-serving opt-in knob, its premium pricing, and lifecycle — described but off by default), serving performance observations such as TTFT/output rate/time-to-answer with source and verification date, serverless-vs-dedicated availability, endpoint/auth/extra-header metadata, provider healthcheck probes, and provider caveats;
• provider-catalog.schema.json — JSON Schema for the catalog contract;
• harn-provider-catalog.d.ts — type-only TypeScript declarations for hosts that load provider-catalog.json directly;
• HarnProviderCatalog.swift — Swift Codable types for hosts that load provider-catalog.json directly.

Use the providers command group for the artifact lifecycle:

# Regenerate the embedded TOML snapshot and all checked-in artifacts.
harn provider catalog generate

# Validate logical catalog invariants, JSON Schema compatibility, and
# checked-in artifact drift.
harn provider catalog generate --check

# Run the existing refresh workflow through the command group.
harn provider catalog refresh --check

# Regenerate the checked-in capability matrix docs.
harn provider catalog matrix

# CI gate: compare the capability matrix docs with source fragments.
harn provider catalog matrix --check

Matrix and support generation read the same on-disk catalog and capability fragments as generate. Run them from the repository root; fragment edits don't require rebuilding the CLI. Outside a checkout, missing source directories use embedded defaults. Present but invalid sources fail instead of falling back.

Support generation recommends one route per provider and never picks one by price. A provider with an active chat route needs a recommended_model entry in crates/harn-cli/data/provider_support_notes.toml or a qc_defaults entry; otherwise harn provider catalog support fails and names it. Adding a new provider, or a cheaper row to an existing one, therefore cannot move a recommendation silently.

Matrix generation is hermetic by default. To inspect local coding-agent parity receipts without changing checked-in output, pass them explicitly:

harn provider catalog matrix \
  --empirical .harn-runs/coding-agent-bench/latest/tool_mode_parity_overlay.toml \
  --stdout

make gen-provider-catalog and make check-provider-catalog route through the same source-fragment graph as harn provider catalog generate, so stale embedded snapshots from the running binary cannot influence checked-in output. The full make all gate includes check-provider-catalog, check-provider-matrix, check-provider-support, and the refresh workflow drift gate.

For local or private models, pass a providers-style TOML overlay to harn provider catalog validate --overlay <path> or harn provider catalog export --overlay <path>. export is for explicit overlay artifacts only; checked-in catalog files are generated with harn provider catalog generate. Overlays are merged with the same precedence as runtime provider config, so private providers, aliases, deprecation notes, quality tags, pricing, and transport settings can be validated before they are published.

Overlays can also hide baseline routes that are broken or unsupported in the embedding product. [suppress] removes the model row, its aliases, and any recommendation variant derived from it from the exported and served artifact (runtime resolution of an explicitly requested id is unaffected):

[suppress]
routes = [
  "together:Qwen/Qwen3-Coder-Next-FP8",
  "ollama:qwen3.6:35b-a3b-coding-nvfp4",
]

Selectors are provider:model_id, split on the first colon only, so model ids that themselves contain colons (Ollama image tags) work. Because an overlay's [models] entries replace whole rows, pairing a new row with a [suppress] entry for the old id also expresses a route rename — no post-export patching required.

To adjust a single field of a baseline row without copying the whole row (and silently freezing its other fields against catalog updates), use a field-wise [patch.models.<id>] patch instead of a [models.<id>] replacement — see "Field-wise catalog patches" in the providers guide:

[patch.models."deepinfra/openai/gpt-oss-120b"]
stream_timeout = 1200.0

harn provider catalog overlay-audit --overlay <file> --check finds the entries that have already drifted into whole-row copies or stopped saying anything at all. Run it from the embedding product's CI, with the Harn release that product pins — see "Auditing an overlay" in the providers guide.

Structured capability fields (tool_support, modalities, reasoning, prompt_cache) come from the capability matrix, not from models.*.capabilities tags (legacy, parse-only). For overlay-declared private or local models, pass a matching capabilities overlay (the same layout as the built-in capabilities.toml):

harn provider catalog export --overlay providers.toml \
  --capabilities-overlay capabilities.toml --output-dir out

[[provider.private]]
model_match = "*"
native_tools = true
vision = true
prompt_caching = true

The serve runtime honors the same data via the manifest [capabilities] section, so the exported artifact and the live _harn/providerCatalog / GET /v1/provider-catalog responses agree.

Model picker presentation metadata#

Model-selection UIs should render from [presentation] records instead of grouping model IDs with host-side string rules. Recommendation variants are stable keyed rows with authored copy and one small selector vocabulary:

[presentation.variants.balanced]
order = 20
label = "Balanced"
description = "Default cost and capability trade-off for everyday work."
selector = { kind = "alias", name = "mid" }

Selectors may be alias, model, best_local, cheapest_hosted, largest_vision_context, or largest_context. The exported variants array contains only the resolved provider and model, so clients do not implement the selector logic.

Families describe one- or two-dimensional pickers. Use a model dimension when values select concrete model IDs, and a reasoning_effort dimension when values select provider effort tokens:

[presentation.families.example]
label = "Example family"
plain_description = "Choose a model size and how much time it spends reasoning."

[[presentation.families.example.dimensions]]
key = "variant"
label = "Model size"
plain_description = "Larger variants trade speed and cost for capability."
kind = "model"
ordered_values = [
  { value = "small", label = "Small", plain_description = "Fast and inexpensive.", relative_cost_hint = 1, relative_speed_hint = 5, model_id = "example-small" },
  { value = "large", label = "Large", plain_description = "Most capable.", relative_cost_hint = 5, relative_speed_hint = 1, model_id = "example-large" },
]

[[presentation.families.example.dimensions]]
key = "effort"
label = "Thinking time"
plain_description = "More thinking can improve difficult answers."
kind = "reasoning_effort"
ordered_values = [
  { value = "low", label = "Low", plain_description = "Brief reasoning.", relative_cost_hint = 1, relative_speed_hint = 5 },
  { value = "high", label = "High", plain_description = "Deeper reasoning.", relative_cost_hint = 4, relative_speed_hint = 2 },
]

[[presentation.families.example.presets]]
id = "balanced"
label = "Balanced"
plain_blurb = "A practical default for everyday work."
coordinates = { variant = "small", effort = "high" }

For a one-dimensional effort family, set model_id on the family and omit a model dimension. Presentation-family overlays replace a same-ID family whole; ordered dimensions and presets are not merged element by element.

Every referenced model must belong to one provider. Effort availability is not duplicated in the presentation row: the artifact projects each model's capability-owned reasoning.effort_levels, and a host enables a grid cell only when the selected model contains that effort token. Presets must resolve to a valid cell. Add a one-sentence blurb to each concrete model row so the detail view can explain its trade-off without marketing copy.

The exported families array preserves dimension and preset order. The flat models array remains available for scripting and backward-compatible lists.

Runtime surfaces#

Thin clients should prefer the live harn-serve catalog when a runtime is available:

• REST: GET /v1/provider-catalog
• ACP: JSON-RPC request method _harn/providerCatalog

Both return the same provider catalog v8 artifact shape: schema_version, schema, generated_by, providers, models, aliases, variants, families, routing_routes, and qc_defaults. Each model owns a non-empty display_name for compact status surfaces alongside its full name identity. Each provider's cache_usage_accounting distinguishes a verified zero-hit cache report from a route that does not expose cache usage. The response is already normalized through the server's effective provider and capability overlays, so clients can render model pickers, key requirements, aliases, local/cloud grouping, context windows, tool support, regional endpoint selectors (endpoint.region_env plus endpoint.regions), presentation families/presets, and pricing without shipping their own model/provider tables. routing_routes is the host-trusted route-decision projection for products that need provider/model/family/capability/timeout rows without resolving live secret values.

When harn-serve is not running, clients can still use the checked-in spec/provider-catalog/provider-catalog.json artifact as a bundled baseline. Product and user overlays should be composed in this order, with later layers winning per key:

1. Harn's bundled provider catalog (crates/harn-provider-catalog/src/providers.toml, generated from crates/harn-vm/src/llm/catalog_sources/, or the generated provider-catalog.json when the client cannot load TOML directly)
2. product or managed providers.toml overlay (HARN_HOST_PROVIDERS_CONFIG for an embedding host)
3. user-global providers.toml (HARN_PROVIDERS_CONFIG or ~/.config/harn/providers.toml)
4. workspace/package [llm] tables for the current run

Once a client has fetched GET /v1/provider-catalog or _harn/providerCatalog, it should treat that response as the effective catalog and avoid applying client-side model/provider patches on top.

Runtime refresh#

Harn can also install a validated runtime overlay on top of the bundled catalog without rebuilding the binary:

fn main(harness: Harness) {
  const report = harness.llm.catalog_refresh()
  harness.stdio.println(to_string(report.status))
}

The same primitive is available as the free builtin llm_catalog_refresh(options?) for scripts that do not receive a Harness. options.url overrides the source URL and options.force ignores a fresh cache entry. The default source is https://harnlang.com/provider-catalog/provider-catalog.json; set HARN_PROVIDER_CATALOG_URL to point at a different catalog.

Refresh behavior is intentionally fail-closed:

• HARN_DISABLE_CATALOG_REFRESH=1 skips refresh and keeps the bundled baseline.
• Remote documents are deserialized against the generated provider catalog contract and then checked with the same logical validator used by harn provider catalog validate.
• Signed envelopes use an Ed25519 signature over the canonical catalog JSON. Configure trusted keys with HARN_PROVIDER_CATALOG_TRUSTED_KEYS=key_id=base64_public_key.
• Unsigned documents are accepted only from loopback development URLs, or when HARN_PROVIDER_CATALOG_ALLOW_UNSIGNED=1 is explicitly set.
• Valid catalogs are cached under $HARN_STATE_DIR/cache/provider-catalog/ with their ETag and TTL. Network failures or malformed documents fall back to a valid cached catalog when one exists, otherwise to the bundled baseline.
• Refresh is skipped inside a live agent_loop; call it before entering the loop so model selection stays deterministic for the run.

harn provider catalog show --refresh runs the same refresh path before printing the catalog. ACP model selectors read the merged runtime catalog, so newly refreshed model IDs appear in clients without regenerating Swift or TypeScript code.

Architecture#

Three layers, kept deliberately small so other repos can extend or replace them without forking the entry script.

Pure logic: scripts/provider_catalog_refresh.harn#

• observation(provider, model_id, fields, provenance) — constructor for the per-source observation record. Every observation carries the source URL, kind (html / json / toml), owner (provider / aggregator), observed_at, fetched_at, confidence (high / medium / low), requires_key, and optional terms_notes.
• normalize_observations(raw) — reduces overlapping observations for the same (provider, model_id) into one canonical record. Provider-owned sources win on conflicting numeric or capability claims; aggregator-owned sources fill gaps. Returns the canonical list plus a conflict log surfaced in the report.
• build_drift(observations, catalog) — compares observations to a catalog dict (live harness.llm.provider_catalog() for --live, the bundled fixture for --check) and returns {added, removed, changed, unknown_pricing, low_confidence, requires_key}. Removals only fire when at least one source for that provider successfully reported, so a missing API key never silently looks like a removal. Missing price fields mean “not reported,” not “removed.” Stored dated promotions are compared by their dates and rates.
• render_markdown_report(drift, conflicts, adapter_runs, meta) — renders the section-by-section markdown report (adapter status, added/removed/changed models, conflicts, unknown pricing, low-confidence claims, key-required observations).
• render_candidate_toml(observations, meta) — renders the review-ready TOML fragment.

Source adapters: scripts/provider_catalog_sources.harn#

Each adapter is a function adapter(env, config) -> {run, observations}.

• html_pricing_table_adapter(env, config) — extracts a pricing / context-window table from an HTML source via the std/web builtins. config.column_map maps HTML header text to observation fields, so the adapter handles different page layouts without open-coded HTML parsing.
• json_api_adapter(env, config) — fetches a JSON endpoint and runs a config.mapper closure to extract model records.
• key_required_adapter(env, config) — wraps another adapter and gates it on config.key_envs. When no required env var is present, the adapter records status: "skipped" and lists the gating env names in the report instead of pretending the source returned empty.

Entry script: scripts/update_provider_catalog.harn#

Fixture mode replays Anthropic and OpenAI pricing pages, the OpenRouter public index, and a key-gated Fireworks source. Live mode reads model-index sources and its network allowlist from spec/provider-catalog-refresh-sources.json, including the public OpenRouter index. Each source selects an existing mapper by a closed, typed name and declares whether its index requires a key and who owns its observations.

The provider catalog supplies credential environment names and authentication style. The source registry carries no duplicate credential aliases. Harn's VM catalog projects these facts from the same artifact used by catalog tooling and includes the provider's local or hosted classification.

Both modes validate the source registry before running an adapter. Empty registries, duplicate source IDs, unknown providers, and malformed typed records fail the refresh, including the offline --check path.

Provenance contract#

Each observation carries:

After normalize_observations, the merged record adds a sources list that retains every contributing source so reviewers can see which page or API claimed which value.

Scheduling#

The workflow runs as an ordinary Harn script. To put it on a weekly cadence, register it as a cron trigger with the standard harn-orchestrator cadence:

# harn.toml in a project repo
[[triggers]]
id = "weekly-provider-catalog-refresh"
kind = "cron"
provider = "cron"
schedule = "0 9 * * MON"
timezone = "Etc/UTC"
match = { events = ["cron.tick"] }
handler = "refresh_provider_catalog"
budget = { daily_cost_usd = 0.10, max_concurrent = 1 }

The handler shells out to harn run scripts/update_provider_catalog.harn -- --live and posts the generated markdown report to wherever your team reviews catalog updates (Slack, Linear, PR description). The workflow never publishes catalog changes on its own.