# Otis

> Evals infrastructure for complex agents

Trust every change you ship to your agent.

Otis builds your agent’s own living benchmark from fully redacted product
telemetry. Simulate production to verify every improvement drives user value.

The same copy, laid out as the page reads, is at
https://www.runotis.com/llms.md

## Evals that cover your users’ every need.

On the page, each step plays a short product demo for an example company. The
conversations and figures in those demos are illustrative, not real data.

1. Find your eval blindspots. Otis analyzes every conversation for what the
   user was trying to do and whether they succeeded. The signals and edge cases
   Otis extracts show which coverage gaps and scoring issues in your eval suite
   to fix first.
2. Build test cases from real agent runs. For every coverage gap, Otis writes
   cases from real conversations: recorded assertions, tool side-effects, or
   LLM judges tuned on what users did next.
3. High-fidelity replays, before your change ships. Replay complex agents in a
   world built from production traces, never touching prod. Smoke-test any PR
   or local change against holdout cases your coding agent never sees.
4. Check your changes deliver for real users. Ship to prod or run an
   experiment, and Otis ensures its predictions translate to real gains.

## A blueprint for improving agents

- Product — Specs: describe an agent feature without coding it. Sketch a
  feature. Otis turns it into checks layered on your benchmark, then picks
  production traces you annotate to guide desired behavior.
- Engineering — Hill-climb your way to Product’s ask. Your coding agent tunes
  the prompt and harness to the spec, graded against a holdout set. When gains
  stall, Otis sorts what still fails by cause. Where rubrics disagree, real
  user behavior decides.
- Subject matter experts — Loop in reviewers without leaving your flow.
  Delegate and track annotation work through review workflows, including
  custom UIs. Each label tunes the judge, and each disagreement becomes a test
  case.

## Your agent’s world is rich in context. Otis lets you leverage it.

- Understand outcomes: Otis connects outcomes across UI, agents, and MCP. It
  records when an agent creates a design that is shared, or makes code edits
  that are thrown away.
- See the big picture: Otis takes a wide-angle view of user context: when they
  joined, their account plan, how they’ve used the product, and what they’re
  trying to accomplish.
- Balance cost and value: Only some users and tasks are worth spending tokens
  on. Otis knows if a task’s success tends to lead to conversion, retention or
  churn.

## Always Private. Secure. Compliant.

- Privacy: Customizable PII redaction · No browser session replay · ZDR at
  model providers
- Compliance: SOC 2 Type II (audit period) · GDPR compliant · HIPAA compliant
- Security: Granular permissions · SSO / SAML · Audit logs

## Build better agents.

Now in production with select design partners.

## Questions

### What do you mean by benchmark?

Benchmarks are reproducible ways of measuring quality or performance against a
fixed set of tests in a well-known environment. Most benchmarks are public, and
change rarely, but they don’t have to.

In Otis, benchmarks are *private* to your organization, *living* in the sense
that they evolve as usage and your codebase changes, and *versioned*. Otis
builds benchmarks to give you a way of testing the quality of your agent. They
are representative of the patterns and distribution of your real users, as well
as errors, issues, and edge cases.

### How’s that different from a set of traces?

Otis leverages broad context of your product and its users to understand where
your agent contributes to user value.

Otis stitches together activity across UI, MCP, CLI and agents. It sees errors,
implicit frictions, and explicit feedback signals. It understands conversion,
retention, and churn.

Otis builds a picture of user intent and task outcomes. By tracking what users
do with the outputs of agent runs in the context of your unique product, Otis
can grade those runs.

### Can’t I just hand my traces to my coding agent?

Coding agents can turn failures in individual traces into test cases. Otis
hands your coding agent the most relevant traces, representative of all usage,
fully redacted of PII and complete with valuable context. This ensures you can
identify and close the coverage gaps in your current evals.

### How do I give Otis access to my telemetry?

After you add the Otis MCP server and skills to your coding agent, Otis can
install the Otis SDK, instrument your codebase, and raise a PR for review.
Telemetry starts arriving once you merge and deploy.

The SDK runs in Node.js, in serverless runtimes and in the browser. It traces
calls made through every popular agent framework, harness, model SDK and AI
gateway. It works seamlessly alongside observability tools.

Otis also supports you sending OpenTelemetry (OTLP) format spans directly, for
example from OpenRouter. You can also get started by uploading exports of
OTLP-format spans from other tools.

### What do you do to respect user privacy?

Before any telemetry hits disk, Otis removes sensitive data using custom
language models. Every identifier is replaced with invented, consistent
stand-ins. IDs are pseudo-anonymized.

Admins can audit and customize PII controls.

Otis ensures any use of models is done with Zero Data Retention (ZDR)
agreements in place, so your data is never used to train public models.

### How does Otis integrate into my workflow?

Otis works where your team already works, in your desktop agent or coding
agent. You can access it in Slack, or ask Otis a question in the Otis app.
Connecting GitHub, Linear and other tools lets you integrate Otis further into
your team’s existing processes.

### How is Otis related to post-training?

Otis benchmarks are great for improving agent code, tools, and harnesses. They
are very useful for evaluating the effect of switching model. Projects
typically find substantial gains here before considering post-training, which
is complementary.

Post-training is tuning an open-source model’s weights to improve it
specifically for your product. The telemetry Otis produces, including outcomes,
is a valuable input to that process.

### Who’s using Otis right now?

Otis is working with AI-native companies building and operating complex agents
in production, ranging from startups to enterprises. Please
[get in touch](mailto:hello@runotis.com) to discuss your needs or
[join the waitlist](https://app.runotis.com/waitlist) to get access when the
product is more broadly available.

Learn more about Otis with one of our founding team.

© 2026 Ortelis Inc

## Links

- Full page copy: https://www.runotis.com/llms.md
- Join waitlist: https://app.runotis.com/waitlist
- Get in touch: mailto:hello@runotis.com
- Book a call: https://cal.com/team/runotis/otis-call
- Josh Greene: https://www.linkedin.com/in/greenejoshua/
- Tim Moreton: https://www.linkedin.com/in/timmoreton/
- Amy Slawson: https://www.linkedin.com/in/amyslawson/
- Privacy: https://www.runotis.com/privacy
- Terms of service: https://www.runotis.com/terms