# Otis Evals infrastructure for complex agents ## Trust every change you ship to your agent. Otis builds your agent’s own living benchmark from fully redacted product telemetry. Simulate production to verify every improvement drives user value. [Join waitlist](https://app.runotis.com/waitlist) ## Evals that cover your users’ every need. *On the page, each step plays a short product demo for an example company. The conversations and figures in those demos are illustrative, not real data.* ### 01 — Find your eval blindspots. Otis analyzes every conversation for what the user was trying to do and whether they succeeded. The signals and edge cases Otis extracts show which coverage gaps and scoring issues in your eval suite to fix first. ### 02 — Build test cases from real agent runs. For every coverage gap, Otis writes cases from real conversations: recorded assertions, tool side-effects, or LLM judges tuned on what users did next. ### 03 — High-fidelity replays, before your change ships. Replay complex agents in a world built from production traces, never touching prod. Smoke-test any PR or local change against holdout cases your coding agent never sees. ### 04 — Check your changes deliver for real users. Ship to prod or run an experiment, and Otis ensures its predictions translate to real gains. ## A blueprint for improving agents *Three tabs, one per role, each with a product demo.* ### Product — Specs: describe an agent feature without coding it. Sketch a feature. Otis turns it into checks layered on your benchmark, then picks production traces you annotate to guide desired behavior. ### Engineering — Hill-climb your way to Product’s ask. Your coding agent tunes the prompt and harness to the spec, graded against a holdout set. When gains stall, Otis sorts what still fails by cause. Where rubrics disagree, real user behavior decides. ### Subject matter experts — Loop in reviewers without leaving your flow. Delegate and track annotation work through review workflows, including custom UIs. Each label tunes the judge, and each disagreement becomes a test case. ## Your agent’s world is rich in context. Otis lets you leverage it. ### Understand outcomes Otis connects outcomes across UI, agents, and MCP. It records when an agent creates a design that is shared, or makes code edits that are thrown away. ### See the big picture Otis takes a wide-angle view of user context: when they joined, their account plan, how they’ve used the product, and what they’re trying to accomplish. ### Balance cost and value Only some users and tasks are worth spending tokens on. Otis knows if a task’s success tends to lead to conversion, retention or churn. ## Always Private. Secure. Compliant. - **Privacy:** Customizable PII redaction · No browser session replay · ZDR at model providers - **Compliance:** SOC 2 Type II (audit period) · GDPR compliant · HIPAA compliant - **Security:** Granular permissions · SSO / SAML · Audit logs ## Build better agents. Now in production with select design partners. [Join waitlist](https://app.runotis.com/waitlist) ## Questions ### What do you mean by benchmark? Benchmarks are reproducible ways of measuring quality or performance against a fixed set of tests in a well-known environment. Most benchmarks are public, and change rarely, but they don’t have to. In Otis, benchmarks are *private* to your organization, *living* in the sense that they evolve as usage and your codebase changes, and *versioned*. Otis builds benchmarks to give you a way of testing the quality of your agent. They are representative of the patterns and distribution of your real users, as well as errors, issues, and edge cases. ### How’s that different from a set of traces? Otis leverages broad context of your product and its users to understand where your agent contributes to user value. Otis stitches together activity across UI, MCP, CLI and agents. It sees errors, implicit frictions, and explicit feedback signals. It understands conversion, retention, and churn. Otis builds a picture of user intent and task outcomes. By tracking what users do with the outputs of agent runs in the context of your unique product, Otis can grade those runs. ### Can’t I just hand my traces to my coding agent? Coding agents can turn failures in individual traces into test cases. Otis hands your coding agent the most relevant traces, representative of all usage, fully redacted of PII and complete with valuable context. This ensures you can identify and close the coverage gaps in your current evals. ### How do I give Otis access to my telemetry? After you add the Otis MCP server and skills to your coding agent, Otis can install the Otis SDK, instrument your codebase, and raise a PR for review. Telemetry starts arriving once you merge and deploy. The SDK runs in Node.js, in serverless runtimes and in the browser. It traces calls made through every popular agent framework, harness, model SDK and AI gateway. It works seamlessly alongside observability tools. Otis also supports you sending OpenTelemetry (OTLP) format spans directly, for example from OpenRouter. You can also get started by uploading exports of OTLP-format spans from other tools. ### What do you do to respect user privacy? Before any telemetry hits disk, Otis removes sensitive data using custom language models. Every identifier is replaced with invented, consistent stand-ins. IDs are pseudo-anonymized. Admins can audit and customize PII controls. Otis ensures any use of models is done with Zero Data Retention (ZDR) agreements in place, so your data is never used to train public models. ### How does Otis integrate into my workflow? Otis works where your team already works, in your desktop agent or coding agent. You can access it in Slack, or ask Otis a question in the Otis app. Connecting GitHub, Linear and other tools lets you integrate Otis further into your team’s existing processes. ### How is Otis related to post-training? Otis benchmarks are great for improving agent code, tools, and harnesses. They are very useful for evaluating the effect of switching model. Projects typically find substantial gains here before considering post-training, which is complementary. Post-training is tuning an open-source model’s weights to improve it specifically for your product. The telemetry Otis produces, including outcomes, is a valuable input to that process. ### Who’s using Otis right now? Otis is working with AI-native companies building and operating complex agents in production, ranging from startups to enterprises. Please [get in touch](mailto:hello@runotis.com) to discuss your needs or [join the waitlist](https://app.runotis.com/waitlist) to get access when the product is more broadly available. ## Learn more about Otis with one of our founding team. - [Josh Greene](https://www.linkedin.com/in/greenejoshua/) - [Tim Moreton](https://www.linkedin.com/in/timmoreton/) - [Amy Slawson](https://www.linkedin.com/in/amyslawson/) [Book a call](https://cal.com/team/runotis/otis-call) --- © 2026 Ortelis Inc · [Privacy](https://www.runotis.com/privacy) · [Terms of service](https://www.runotis.com/terms)