# Tutorial: build an eval pipeline

> This tutorial builds a small evaluation loop that runs a set of examples, records metrics, and produces an auditable summary. The goal is to make quality visible, not to build...

Website: https://harnlang.com/tutorial-eval-pipeline.html

This page documents Harn, which is pre-1.0. Language, standard library, and CLI APIs may change. If the intended version is unclear, clarify before using this page.

---

This tutorial builds a small evaluation loop that runs a set of examples,
records metrics, and produces an auditable summary. The goal is to make quality
visible, not to build an elaborate benchmark harness.

Use the companion example as a baseline:

```bash
harn run examples/eval-workflow.harn
```

## 1. Define the dataset inline

Start with a tiny set of representative inputs. Keep the examples small enough
that you can inspect failures by eye:

```harn
pipeline main(harness: Harness) {
  const cases = [
    {id: "case-1", input: "What is 2 + 2?", expected: "4"},
    {id: "case-2", input: "Capital of France?", expected: "Paris"},
    {id: "case-3", input: "Color of grass?", expected: "green"},
  ]

  harness.stdio.log("Loaded ${cases.count} eval cases")
}
```

## 2. Run the cases in parallel

If each case is independent, use `parallel each` so the slow parts overlap.

```harn
pipeline main(harness: Harness) {
  const cases = [
    {id: "case-1", input: "What is 2 + 2?", expected: "4"},
    {id: "case-2", input: "Capital of France?", expected: "Paris"},
    {id: "case-3", input: "Color of grass?", expected: "green"},
  ]

  const results = parallel each cases { tc ->
    const answer = harness.llm.call(
      tc.input, "Answer in one word or short phrase.", {
        temperature: 0.0,
        max_tokens: 64,
      })

    {
      id: tc.id,
      expected: tc.expected,
      actual: answer.text,
      correct: answer.text.contains(tc.expected),
    }
  }

  harness.stdio.log(json_stringify(results))
}
```

For a real eval suite, replace the inline `cases` list with a manifest or a
dataset file that your pipeline reads with `harness.fs.read_text()`.

## 3. Record metrics

The important part of an eval pipeline is the metric trail. Use
`eval_metric()` to record per-case and aggregate results.

```harn
pipeline main(harness: Harness) {
  const cases = [
    {id: "case-1", input: "What is 2 + 2?", expected: "4"},
    {id: "case-2", input: "Capital of France?", expected: "Paris"},
  ]

  let passed = 0
  for tc in cases {
    const answer = harness.llm.call(
      tc.input, "Answer in one word.", {temperature: 0.0},
    )
    const correct = answer.text.contains(tc.expected)
    if correct {
      passed = passed + 1
    }
    eval_metric("case_correct", correct, {case_id: tc.id})
  }

  // `/` on two ints truncates: 1 of 2 passing would report 0, not 0.5.
  const accuracy = to_float(passed) / cases.count
  eval_metric("accuracy", accuracy, {passed: passed, total: cases.count})
  eval_metric("run_id", harness.random.uuid())
  eval_metric("generated_at", harness.clock.timestamp())
}
```

## 4. Export a report

Once the metrics are recorded, write a compact report so a later run can diff
the results.

```harn
pipeline main(harness: Harness) {
  const summary = {
    run_id: harness.random.uuid(),
    generated_at: harness.clock.timestamp(),
    accuracy: 0.83,
    notes: "Replace the fixed accuracy with real case scoring",
  }

  harness.fs.write_text("eval-summary.json", json_stringify(summary))
  harness.stdio.log(json_stringify(summary))
}
```

## 5. How to use it

Run the pipeline and read the metric trail it prints:

```bash
harn run examples/eval-workflow.harn
```

```text
[harn] === Recorded Metrics ===
[harn]   case_correct = true
[harn]   case_correct = true
[harn]   case_correct = true
[harn]   accuracy = 1.0
[harn]   test_suite_size = 3
```

To compare runs over time you need a saved run record. `harn run` on a plain
pipeline does not write one — `.harn-runs/` is populated by `workflow_execute`,
and that is what `harn eval` and `harn replay` consume:

```bash
harn eval .harn-runs/<run-id>.json
harn eval .harn-runs/                 # every record in the directory
```

See [the workflow runtime](./workflow-runtime.md) for the run-record shape and
how to produce one.

A good eval pipeline answers three questions:

- did the model improve?
- did latency or token usage regress?
- which cases failed, and why?

## Skill and guidance gates

Use `harn eval skill-gate` when the artifact under review is a skill or
guidance edit. The manifest records contamination-safe held-out tasks,
with-vs-without observations, the frontier score, context-cost inputs, and
immutable grader checksums:

```bash
harn eval skill-gate examples/evals/skill-gate/smoke/manifest.json \
  --output .harn-runs/skill-gate/smoke
```

The command writes `summary.json`, `per_case.jsonl`, `summary.md`, and a
machine-readable `receipt.json` using `harn.skill_gate.receipt.v1`. The gate
excludes static public/pre-cutoff tasks, reports gap recovery per cluster,
rejects regressions and context bloat, and fails closed when a protected grader
file or directory hash changes.

---

## Read next

- [Tutorial: MCP server](https://harnlang.com/tutorial-mcp-server.md)
- [Tutorial: durable daemon agent](https://harnlang.com/tutorial-daemon-agent.md)