What do you mean by benchmark?
Benchmarks are reproducible ways of measuring quality or performance against a fixed set of tests in a well-known environment. Most benchmarks are public, and change rarely, but they don’t have to.
In Otis, benchmarks are private to your organization, living in the sense that they evolve as usage and your codebase changes, and versioned. Otis builds benchmarks to give you a way of testing the quality of your agent. They are representative of the patterns and distribution of your real users, as well as errors, issues, and edge cases.
How’s that different from a set of traces?
Otis leverages broad context of your product and its users to understand where your agent contributes to user value.
Otis stitches together activity across UI, MCP, CLI and agents. It sees errors, implicit frictions, and explicit feedback signals. It understands conversion, retention, and churn.
Otis builds a picture of user intent and task outcomes. By tracking what users do with the outputs of agent runs in the context of your unique product, Otis can grade those runs.
Can’t I just hand my traces to my coding agent?
Coding agents can turn failures in individual traces into test cases. Otis hands your coding agent the most relevant traces, representative of all usage, fully redacted of PII and complete with valuable context. This ensures you can identify and close the coverage gaps in your current evals.
How do I give Otis access to my telemetry?
After you add the Otis MCP server and skills to your coding agent, Otis can install the Otis SDK, instrument your codebase, and raise a PR for review. Telemetry starts arriving once you merge and deploy.
The SDK runs in Node.js, in serverless runtimes and in the browser. It traces calls made through every popular agent framework, harness, model SDK and AI gateway. It works seamlessly alongside observability tools.
Otis also supports you sending OpenTelemetry (OTLP) format spans directly, for example from OpenRouter. You can also get started by uploading exports of OTLP-format spans from other tools.
What do you do to respect user privacy?
Before any telemetry hits disk, Otis removes sensitive data using custom language models. Every identifier is replaced with invented, consistent stand-ins. IDs are pseudo-anonymized.
Admins can audit and customize PII controls.
Otis ensures any use of models is done with Zero Data Retention (ZDR) agreements in place, so your data is never used to train public models.
How does Otis integrate into my workflow?
Otis works where your team already works, in your desktop agent or coding agent. You can access it in Slack, or ask Otis a question in the Otis app. Connecting GitHub, Linear and other tools lets you integrate Otis further into your team’s existing processes.
How is Otis related to post-training?
Otis benchmarks are great for improving agent code, tools, and harnesses. They are very useful for evaluating the effect of switching model. Projects typically find substantial gains here before considering post-training, which is complementary.
Post-training is tuning an open-source model’s weights to improve it specifically for your product. The telemetry Otis produces, including outcomes, is a valuable input to that process.
Who’s using Otis right now?
Otis is working with AI-native companies building and operating complex agents in production, ranging from startups to enterprises. Please get in touch to discuss your needs or join the waitlist to get access when the product is more broadly available.
