Bench mode

@runotis/sdk/bench re-runs one recorded case from your production benchmark against your agent's current code. Every model call is live, with your prompts, your model and your keys, because that is what is under test. Every tool call passes through the seam the SDK already holds, where wrap(ai) and tracedTool wrap your tool functions, and is answered by the world before your function would run. Your agent's code does not change; the CI package calls runCase per case and repeat.

It needs three things to exist first:

• A benchmark version pinned in your repository, as otis-benchmark.lock.json, and the cases it names downloaded by the CI package under a run token.
• A manifest, otis-benchmark.json, beside the lockfile: the facts about your code that bench mode reads. The otis-instrument skill drafts it.
• An entry point: your agent as a function from a message list to a turn, named in the manifest.

Install

npm install @runotis/sdk

import { readFile } from "node:fs/promises";
import { runCase, readBenchmarkManifest, readBenchmarkRecording } from "@runotis/sdk/bench";
import { runTurn, readEndState } from "./src/agent/bench";

const manifest = readBenchmarkManifest(JSON.parse(await readFile("otis-benchmark.json", "utf8")));
const recording = readBenchmarkRecording(JSON.parse(await readFile(caseFile, "utf8")));

const result = await runCase(otis, {
  manifest,
  recording,
  run: { runId, lockfileVersion: lockfile.benchmark.versionId },
  entry: runTurn,
  endState: readEndState,
});

otis is the instance initOtis returned, started with the bench credential the run was opened under. Bench mode is on only inside runCase: outside it the seams call your functions as they do in production.

The entry point

A function (input, context) => Promise<unknown>. input.messages is the recording's conversation prefix and the turn the case is cut at, in the AI SDK's ModelMessage form (roles system, user, assistant and tool; parts text, tool-call and tool-result), so an agent on the AI SDK passes it through and another client converts. input.trigger is a headless run's raw input beside its rendering as the first message. input.case carries the case id and repeat. context.signal is an abort signal the world raises when it refuses a call, so an agent that honours it ends the turn there.

export async function runTurn(input, { signal }) {
  return generateText({ model, system, messages: input.messages, tools, abortSignal: signal });
}

Most agents are HTTP routes that load auth, a session and database state first. Giving Otis a function is the adoption cost of bench mode, and the eval-writing workflow is where it is paid.

The manifest

otis-benchmark.json holds facts about the code and nothing a team tunes:

{
  "manifestVersion": 1,
  "entry": { "module": "./src/agent/bench.ts", "export": "runTurn" },
  "endState": { "module": "./src/agent/bench.ts", "export": "readEndState" },
  "harness": ["src/agent/**", "prompts/**"],
  "tools": {
    "lookupOrder": { "kind": "read", "entities": [{ "type": "order", "idArg": "orderId" }] },
    "searchDocs":  { "kind": "retrieval", "index": { "snapshot": "docs-2026-11-01" } },
    "refundOrder": { "kind": "write", "agency": "confirm", "entities": [{ "type": "order", "idArg": "orderId" }] },
    "chargeCard":  { "kind": "never_fakeable" },
    "sendEmail":   { "kind": "never_fakeable" }
  }
}

A tool the manifest does not name is read or write by its name's leading verb and is never refused: a run refuses only what the file says.

How a call is answered

In this order, by the tool's kind:

1. Refused. A never-fakeable tool. The call is classified against the recording as matched (the same tool with the same arguments), different or new, and the agent gets an OtisBenchRefusedError as the tool's error.
2. Live. A retrieval tool with a pinned snapshot, or a live tool: your function runs.
3. The ledger. A write records the fields it wrote under the entities its arguments name; a later read of an entity the case wrote is served from the recorded answer with those fields laid over.
4. Captured. The first unconsumed recorded call with the same arguments, in call order, so a recorded failure then a success replays as such.
5. Approximate. The same after identifiers, timestamps, cursors and page tokens are stripped, then the nearest recorded call to the tool above a floor.
6. Authored. A fixture the run supplied, on a spec's cases.
7. Remote. Otis's own answer, from calls recorded across your whole corpus or generated under the tool's response shape, where the run has a client for it.
8. Miss. Nothing answered: the agent gets an OtisBenchMissError as the tool's error. Your function is never called instead.

Every tool span says which: otis.bench.tier, with otis.bench.refusal on a refused call and otis.bench.recorded_index where a recorded call answered. A refusal and a miss are the world's, not the tool's, so neither counts as a tool failure when the run is graded.

The end-state read

Optional. A function the manifest names under endState, called once after the turn with every entity the case's calls named and whether the case wrote it. It returns each entity's fields after the run, scalars only, read from your own store where you restored one:

export async function readEndState(touched, { signal }) {
  return Promise.all(touched.map(async ({ type, id }) => ({ type, id, fields: await store.scalarsOf(type, id) })));
}

The read lands on a span beside the ledger's final state, and Otis grades a case's end-state assertions against your read where you supplied one and against the ledger otherwise. A read that throws, or runs past endStateTimeoutMs (30 seconds by default), leaves the case without one.

Resolving the lockfile

The cases a run replays come from Otis, not from the repository. resolveLockfile reads otis-benchmark.lock.json, fetches the pinned version's case records and each live case's recording under the bench credential, verifies every recording against the lockfile's content hash before it writes anything, and writes files under a directory of your choosing. It never prints: every failure is a line in its result, and no line carries a word of case text.

import { readFile } from "node:fs/promises";
import { readBenchmarkLockfile, resolveLockfile } from "@runotis/sdk/bench";

const lockfile = readBenchmarkLockfile(JSON.parse(await readFile("otis-benchmark.lock.json", "utf8")));
const resolved = await resolveLockfile(lockfile, { outDir: ".otis-bench" });
if (!resolved.ok) throw new Error(resolved.errors.join("\n"));

What it writes under outDir:

A case retired since the lockfile was written gets its record and no recording, and is listed in retired. The bench credential comes from OTIS_BENCH_KEY and the app from OTIS_APP_URL, as the Vitest reporter reads them; the credential needs the cases:read scope, which the GitHub Actions exchange grants only when the project allowed it.

The harness hash and the pre-run reads

Before the first case runs, the CI package reads the harness: the files your manifest's harness globs, each nested agent's globs and the entry module name, and the installed versions of the common harness packages (ai, @ai-sdk/*, @anthropic-ai/sdk, @anthropic-ai/claude-agent-sdk, openai, @modelcontextprotocol/sdk, @mastra/core, @langchain/*, plus your manifest's harnessPackages). preRunRead hashes them into the harness hash a run is keyed on: when it matches the cached baseline's, the run short-circuits to the baseline's result; when it moves, the run runs. No commit is in it, so a change to code alone runs, and the Otis SDK's own version is left out, so an SDK upgrade moves no baseline.

import { preRunRead } from "@runotis/sdk/bench";

const read = await preRunRead({ manifest, rootDir: process.cwd(), cases: resolved.cases });
read.harnessHash;          // sha256:… over the recipe below
read.pathDigests;          // per harness file, so a moved hash can name the paths that moved
read.packages;             // the installed versions in the hash
read.caseText.matches;     // cases whose text or surrogates the harness contains, by id and path

The recipe is otis-harness-hash/1, then one line per file (file, the POSIX path relative to the root, the SHA-256 of its bytes, sorted by path), then one line per package (package, name, version, sorted by name), hashed with SHA-256. The glob expansion follows no symbolic link, never enters node_modules or .git, and skips a pattern that would leave the root.

The same read checks the harness for case text, which a coding agent pasting a failing case into a prompt would put there: a run of at least twelve consecutive words of a case's text (sharedWordsWindow, a setting), or one of the case's surrogate strings standing as a whole word. A matched case is named by id with the harness paths it was found in, and leaves that run's delta; no word of the text is in the result.

After the run, postRunRead pairs what the SDK observed on the cases' spans (observed on each runCase result: the prompt, tool and model hashes, the resolved checkpoint, the run hash) and the share of each tool's calls your own function executed with the baseline, into two labels the comment carries: an incomplete manifest, when the hash held but a prompt, tool definition or model surface moved, so a file the manifest's paths do not cover changed; and changed and not exercised, when the hash moved but nothing the SDK observed did, named by the paths that moved and the nested agent they belong to, so the comment can offer cases cut at that agent's root. Both are labels, never failures.

What the spans carry

Every span the case emits is stamped with the run's identity (otis.bench.run_id, otis.bench.case_id, otis.bench.repeat, otis.bench.lockfile_version) and the case's redaction record (otis.bench.redaction), so Otis routes them to its bench storage and masks exactly the surrogate strings the record names. The SDK's own redactor leaves those strings as they are and redacts everything else, a live retrieval's documents included.

What runCase returns

runCase never posts: the CI package sums results into the pull request comment.

Running in CI

otis bench ci, from @runotis/cli, runs the whole loop: it takes a run token for the job (or a bench key), reads the harness, opens a run, downloads the cases the lockfile pins, runs each due case in bench mode per repeat with every declared store reset first, closes the run, waits for Otis to grade it, opens one escalation run where the grading offers it, and posts one comment and one commit status on the pull request with the job's own token. The loop is runCi in @runotis/sdk/bench; the CLI owns the name and parses the flags.

name: Otis benchmark
on:
  pull_request:
  push:
    branches: [main]
permissions:
  contents: read
  id-token: write        # the identity token the exchange takes; no Otis secret is stored
  statuses: write        # the commit status `otis/bench`
  pull-requests: write   # the comment
jobs:
  bench:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: pnpm/action-setup@v6
      - uses: actions/setup-node@v6
        with: { node-version: '22' }
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec otis bench ci
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}   # your agent's own provider keys

What runs when. A pull request runs the smoke subset. A push to the default branch and a schedule run every case at the full repeats and ask to be the version's baseline, which Otis admits only on the default branch against a frozen world. A manual dispatch runs every case at the full repeats and does not ask, so a version's first baseline comes from its first merge or scheduled run. The label otis-bench:full on a pull request asks for a full run. --kind and --trigger override the event. Outside GitHub Actions, set OTIS_BENCH_KEY to a bench key from the project's settings and pass both flags.

The credential. On GitHub Actions the job stores no Otis secret: the package asks the app where the exchange is (GET /api/bench/exchange), mints the job's identity token for the answered audience, and exchanges it for a run token that lasts the job. The project's settings must have linked the repository and allowed GitHub Actions. Other CI presents a long-lived bench key in OTIS_BENCH_KEY.

The comment. One per pull request, updated on each run: the deterministic tier's counts; the read checks that failed on cases in the blocking set, as a table; the delta per stratum with its interval and unit, marked smoke where a run had no interval, with an escalation's rows replacing the smoke rows for its strata; the three failing cases worth reading, by id, stratum, first failing check and tier; consistency checks not yet audited against your own labels, per case; and the reads around the run: case text found in the harness (by file path and case id; those cases left the delta), a change the run never exercised, a surface that moved while the harness hash held, a retrieval tool with no pinned snapshot, and the baseline's admission. Nothing in it is a case's text. It is also written to .otis-bench/comment.md; put .otis-bench/ in .gitignore.

The status and the exit code. The commit status otis/bench can be required by a branch protection rule. It is failure only when a read check failed on a case in the blocking set; success otherwise once graded; error when the run did not complete, or when grading had not finished after the wait (twenty minutes by default). A grading that outlives the wait ends as error rather than pending, because nothing updates that status once the job has exited, and a required pending status would block the merge for good. When the job took its run token through the exchange, the package then hands the run off, and when grading finishes Otis's GitHub App posts the result on the pull request as a comment of its own; under a long-lived key, re-run the job with a longer --wait. A judge's verdict never fails it. The exit code is 0 for success, cached or a skipped fork, 1 for a blocking failure, 2 for a run that did not complete or whose grading outlived the wait.

Pull requests from forks. The package runs your repository's code (the entry point, the end-state read and every store reset) while it holds a run token that can read case text, so it never runs a fork's code with a credential. Use the pull_request event: a pull request from a fork gets neither secrets nor an identity token there, and the package skips it with exit 0 and posts nothing. On pull_request_target or workflow_run, which hold your repository's secrets and write token whatever the job checked out, a fork's code is refused. A pull request from one of your own branches runs on pull_request_target only with --allow-pull-request-target.

The Otis benchmark check. Where your organization's installation of the Otis GitHub App has approved the Checks permission, Otis also posts a check run named Otis benchmark on the commit the job tested, once the run is graded, whatever the job did meanwhile. It follows the same rule as the status (failure only on a read check failed on a blocking case, never on a judge's verdict) and carries the counts, the blocking failures and the delta per stratum. A branch protection rule can require it instead of otis/bench. It is posted only for a run on GitHub Actions that took its run token through the exchange, since only that run's repository and commit are verified; a run under a long-lived bench key relies on the job's own status.

When the harness is unchanged. The package hashes the harness before anything runs. When the hash matches the version's cached baseline, nothing runs: the status is set from the baseline and no comment is posted.

Isolation channels. Every run of a case should start from state of its own, which is what stores declares. On your first CI run, on the first run of each new benchmark version, and on every run while a channel is open, the package first runs one due case twice in a row without resetting anything between. If a tool your own code answers (a live or retrieval tool) returns, in the second run, an id the first run's write created, that tool is an isolation channel: some store it reads carries one run's work into the next. The comment names it and labels the run's deltas, and the project's Benchmark settings tab lists it until an admin closes it. Fix it by declaring the store under stores with a reset, then close the channel once a later run shows it re-checked clean. The two probe runs are not graded, enter no delta and count against no cap. Ids are compared by digest inside the job, and the comment and settings name only tools and field names.

runCi(options) from @runotis/sdk/bench takes the same options in a script.