Eval suite verdicts

Post your eval suite's verdicts to Otis after each run, so Otis can set what your judges say against what users did on each kind of request.

@runotis/sdk/vitest is a Vitest reporter that posts an eval suite's verdicts to Otis when a run ends: one verdict per case per scorer. Otis stores them beside the suite's cases and reads them in the grounding report, which compares the suite's pass rate on each kind of request with how users fared on it in production.

It needs two things to exist first:

The suite imported. A coding agent imports the suite with the mapEvalSuite tool of the Otis MCP server, which stores where each case lands on your production tasks. The reporter names each case by the ref that import used.
A bench credential. A run token the GitHub Actions token exchange mints for the job, or a long-lived bench key minted on the project's settings page, in OTIS_BENCH_KEY. An ingest key cannot post verdicts.

Install

npm install @runotis/sdk

Add the reporter beside your usual one in vitest.config.ts, naming the suite as it was imported:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    reporters: ["default", ["@runotis/sdk/vitest", { suite: "evals/support" }]],
  },
});

Set OTIS_BENCH_KEY in the environment the run has. Without it the reporter prints one line and posts nothing, so a local run needs no key.

What is posted

For every test the runner's own verdict: scorer vitest, pass or fail by the test's status, skipped for a test that did not run. For a vitest-evals case the reporter also posts one verdict per judge from the scores the case recorded, with the judge's number, and the case's average as the runner's number.

A judge records no threshold of its own, so its verdict is read against judgeThreshold, which defaults to vitest-evals' default of 1. If your suite sets judgeThreshold, pass the same value:

["@runotis/sdk/vitest", { suite: "evals/support", judgeThreshold: 0.8 }]

A judge's null score is a skipped verdict with no number.

Naming the case

A verdict names its case by ref, the same string the import used for the case's file and name. The default is <file relative to the Vitest root>::<full test name>:

evals/support.eval.ts::support refund flow

Import the suite with refs of that form, or pass refOf to match whatever the import used:

["@runotis/sdk/vitest", {
  suite: "evals/support",
  refOf: (test) => `${test.relativeFile}#${test.title}`,
}]

The reporter's summary lists every ref that matched no imported case, so a mismatch shows in the CI log rather than as missing rows.

Options

Option	Default	What it is
`suite`	required	The suite's name, as imported with `mapEvalSuite`.
`apiKey`	`OTIS_BENCH_KEY`	The bench credential.
`appUrl`	`OTIS_APP_URL`, else `https://app.runotis.com`	The Otis app the verdicts go to.
`runId`	`OTIS_BENCH_RUN_ID`, else the GitHub Actions run and attempt, else a local id	Names the run. A re-post under the same id rewrites its rows; a repeat under another id adds a run.
`commitSha`	`OTIS_BENCH_COMMIT_SHA`, else `GITHUB_SHA`	The commit the run scored.
`judgeThreshold`	`1`	The score a judge passes at.
`refOf`	`<relativeFile>::<fullName>`	How a test names its case.
`rootDir`	Vitest's root	What `relativeFile` is relative to.

Posting a JSON report instead

The same posting takes a report Vitest's JSON reporter wrote (--reporter=json --outputFile=report.json), for a runner that cannot load a reporter or a step that runs after the tests:

import { readFile } from "node:fs/promises";
import { postVitestJsonReport } from "@runotis/sdk/vitest";

const report = JSON.parse(await readFile("report.json", "utf8"));
await postVitestJsonReport(report, { suite: "evals/support", rootDir: process.cwd() });

Posts go to POST /api/bench/scores on the app, at most 1,000 verdicts each, and are retried on a 408, 429 or 5xx and on a connection failure. The reporter never fails the run: every failure is a line in its summary.