Trust every change you ship to your agent.
Otis builds your agent’s own living benchmark
from fully redacted product telemetry.Simulate production to verify every improvement drives user value.
Evals that cover your users’ every need.
Find your eval blindspots.
Otis analyzes every conversation for what the user was trying to do and whether they succeeded.
The signals and edge cases Otis extracts show which coverage gaps and scoring issues in your eval suite to fix first.
What are the gaps in my eval suite?
17% of users are trying to compare plans, one in three fail. You have no test cases for this.
| Task | % of users | Failures | Coverage |
|---|---|---|---|
| Add a section | 9% | 24% | 10% |
| Rearrange the page | 11% | 29% | 25% |
| Compare plans | 17% | 33% | 0% |
| Restyle a component | 18% | 11% | 65% |
| Rewrite product copy | 24% | 8% | 80% |
Want me to show redacted example sessions or write new cases to cover this?
Build test cases from real agent runs.
For every coverage gap, Otis writes cases from real conversations: recorded assertions, tool side-effects, or LLM judges tuned on what users did next.
founder, Pro: “what do I actually get on Teams that I don’t have on Pro? two of us now, hiring a third”
- makes no canvas edits
- names only Teams features that have shipped
- upgrades to Teams within 7 days
freelancer, Pro: “three client storefronts now and each client wants to review previews. is Pro enough for that or do I need Teams?”
recorded: “You’ll need Teams for client review, it’s $49 a seat.”
- makes no tool call
- doesn’t say client review needs Teams
- says what Pro already covers
- states no price
- no downgrade within 30 days
That’s your baseline. The copilot answers plan questions without knowing the designer’s plan, so that’s the fix to make first.
High-fidelity replays, before your change ships.
Replay complex agents in a world built from production traces, never touching prod.
Smoke-test any PR or local change against holdout cases your coding agent never sees.
Opendana-r wants to merge 3 commits into main from fix/plan-context
“Compare plans” gets better. “Add a section” gets worse.
| Task | main | this PR | |
|---|---|---|---|
| Compare plans | 67% | 77% | +10 |
| Add a section | 76% | 74% | −2 |
| Rewrite product copy | 92% | 92% | 0 |
| Cost per resolved task | 9¢ | 9¢ | 0 |
Blocking “Add a section” fell below its 76% floor.
The regression affects free-plan designers most: the agent suggests upgrading mid-edit. See example runs.
@otis fixed in a3f9c1e. Mind running it again?
Check your changes deliver for real users.
Ship to prod or run an experiment, and Otis ensures its predictions translate to real gains.
#488 has been with 10% of designers for five days. They did better.
| Before | After | Predicted | |
|---|---|---|---|
| Compare plans resolved | 67% | 76% | 77% |
| Add a section resolved | 76% | 76% | 76% |
| Upgrades within 7 days | 3.1% | 3.4% | 3.5% |
Compare plans came in a point under my forecast, and nothing else slipped. Widen it to everyone?
A blueprint for improving agents
Specs: describe an agent feature without coding it.
Sketch a feature. Otis turns it into checks layered on your benchmark, then picks production traces you annotate to guide desired behavior.
I want to build a spec for our Otis benchmark for translating a page. Include a judge for whether it reads naturally.
Designers already ask for this: 640 conversations in the last 90 days, mostly German and French. They rewrote the copilot’s translation in 4 of 10.
- Checks
- Translates every string, buttons and alt text included
- Leaves prices, sizes and links as they are
- Judge: reads as if written in that language
- Cases
- 160 generated, on 12 storefront fixtures
- Weight
- Assume 6% of copilot requests
Score today’s copilot on these 160 cases?About $3.40
| Translate a pageHypothetical | 22% |
| All copilot requestsProduction | 74% |
At 6% of requests, today’s copilot would bring the overall score to 71%. Once designers start translating pages, Otis grades both numbers as a prediction.
To check the judge, read 10 conversations this week: 6 where the judge and the designer disagree, and 4 at random. Here’s the first.
AskedTranslate the fall page into German.
JudgeReads naturally
ThenThe designer shortened the headline, which ran off the screen on phones.
Not right. A translation has to fit the layout, on phones too.
Added fits the layout to the spec. 38 of the cases now check it.
Hill-climb your way to Product’s ask.
Your coding agent tunes the prompt and harness to the spec, graded against a holdout set.
When gains stall, Otis sorts what still fails by cause. Where rubrics disagree, real user behavior decides.
The top cause is a rubric question. Engineering’s rubric asks for formal “Sie”, and product’s asks for the tone of the English page.
The 86% rests on German nobody has read yet, so Otis has asked a German reviewer to read it.
Loop in reviewers without leaving your flow.
Delegate and track annotation work through review workflows, including custom UIs.
Each label tunes the judge, and each disagreement becomes a test case.
@Lena Vogt 19 German pages from the copilot need a native speaker’s read. We’re reusing the team’s standard review page but putting the English beside the German. Time estimate: 15 mins
Judge: Reads naturally. Do you agree?
Saved as a test case
Your agent’s world is rich in context.
Otis lets you leverage it.
Understand outcomes
Otis connects outcomes across UI, agents, and MCP. It records when an agent creates a design that is shared, or makes code edits that are thrown away.
See the big picture
Otis takes a wide-angle view of user context: when they joined, their account plan, how they’ve used the product, and what they’re trying to accomplish.
Balance cost and value
Only some users and tasks are worth spending tokens on. Otis knows if a task’s success tends to lead to conversion, retention or churn.
Always Private. Secure. Compliant.
Privacy
- Customizable PII redaction
- No browser session replay
- ZDR at model providers
Compliance
- SOC 2 Type II audit period
- GDPR compliant
- HIPAA compliant option
Security
- Granular permissions
- SSO / SAML option
- Audit logs
Build better agents.
Now in production with select design partners.
Questions
What do you mean by benchmark?
Benchmarks are reproducible ways of measuring quality or performance against a fixed set of tests in a well-known environment. Most benchmarks are public, and change rarely, but they don’t have to.
In Otis, benchmarks are private to your organization, living in the sense that they evolve as usage and your codebase changes, and versioned. Otis builds benchmarks to give you a way of testing the quality of your agent. They are representative of the patterns and distribution of your real users, as well as errors, issues, and edge cases.
How’s that different from a set of traces?
Otis leverages broad context of your product and its users to understand where your agent contributes to user value.
Otis stitches together activity across UI, MCP, CLI and agents. It sees errors, implicit frictions, and explicit feedback signals. It understands conversion, retention, and churn.
Otis builds a picture of user intent and task outcomes. By tracking what users do with the outputs of agent runs in the context of your unique product, Otis can grade those runs.
Can’t I just hand my traces to my coding agent?
Coding agents can turn failures in individual traces into test cases. Otis hands your coding agent the most relevant traces, representative of all usage, fully redacted of PII and complete with valuable context. This ensures you can identify and close the coverage gaps in your current evals.
How do I give Otis access to my telemetry?
After you add the Otis MCP server and skills to your coding agent, Otis can install the Otis SDK, instrument your codebase, and raise a PR for review. Telemetry starts arriving once you merge and deploy.
The SDK runs in Node.js, in serverless runtimes and in the browser. It traces calls made through every popular agent framework, harness, model SDK and AI gateway. It works seamlessly alongside observability tools.
Otis also supports you sending OpenTelemetry (OTLP) format spans directly, for example from OpenRouter. You can also get started by uploading exports of OTLP-format spans from other tools.
What do you do to respect user privacy?
Before any telemetry hits disk, Otis removes sensitive data using custom language models. Every identifier is replaced with invented, consistent stand-ins. IDs are pseudo-anonymized.
Admins can audit and customize PII controls.
Otis ensures any use of models is done with Zero Data Retention (ZDR) agreements in place, so your data is never used to train public models.
How does Otis integrate into my workflow?
Otis works where your team already works, in your desktop agent or coding agent. You can access it in Slack, or ask Otis a question in the Otis app. Connecting GitHub, Linear and other tools lets you integrate Otis further into your team’s existing processes.
How is Otis related to post-training?
Otis benchmarks are great for improving agent code, tools, and harnesses. They are very useful for evaluating the effect of switching model. Projects typically find substantial gains here before considering post-training, which is complementary.
Post-training is tuning an open-source model’s weights to improve it specifically for your product. The telemetry Otis produces, including outcomes, is a valuable input to that process.
Who’s using Otis right now?
Otis is working with AI-native companies building and operating complex agents in production, ranging from startups to enterprises. Please get in touch to discuss your needs or join the waitlist to get access when the product is more broadly available.


