Enterprises are rapidly deploying agents to automate complex work, and every one of them needs a reliable way to simulate, verify, and optimize those agents before trusting them with real work. Litmus gives them that confidence. We learn how a company's people actually work, generate the sandboxes, tasks, and verifiers to match, and evaluate its agents before they ship and once they're in production. The goal is an autonomous research engine that gives every enterprise the kind of internal benchmarking function the frontier labs have.
Litmus was founded by Sidharth Srinivasan, who led AI evals at Box after its CEO recruited him out of Stanford CS. He built the function from inception, running a research engineering team and owning Box's relationships with the frontier labs and data vendors. Litmus has raised $5M from Pear, BoxGroup, Workbench, and Emergence, alongside angels including Aaron Levie (CEO of Box), Arash Ferdowsi (co-founder of Dropbox), and senior leaders at OpenAI, Anthropic, Google, and Mercor. Demand already spans Fortune 15 enterprises, top US banks, AI transformation firms, and AI-native services companies, and we're hiring to keep up.
How we work: in person in FiDi, five days a week. Small team, high ownership, and we care about outcomes, not hours.
Define what "good" means for agents doing real work, and prove it.
The frontier labs know how to measure and improve models for knowledge work: mock the tools, generate tasks from real usage, build verifiers, and audit the results. You'll bring that same rigor to every enterprise's agents, in finance, healthcare, legal, and enterprise operations, where the answer is rarely a simple right or wrong. You'll own the science at the core of Litmus and turn it into something that runs on its own.
What you'll work on:
Verifiers and graders for knowledge work, from calibrated LLM judges to checks grounded in what the agent actually did
Simulated users, environments, and tasks faithful enough that passing them predicts production
Fidelity, judge calibration, and coverage: knowing when you've tested enough, and when a score means something
Golden trajectories, failure taxonomies, and the loop from "here's what broke" to "here's the fix"
Turning what works for one customer into something the product does automatically
Why it's exciting:
You'll write the playbook for measuring agents doing real work, on real agents in high-stakes domains, not leaderboards.
The environments and verifiers that test an agent are exactly what it takes to train one. Measurement is the first step toward agents that improve on their own.
You'll work directly with customers' teams and watch your work change what they ship.
You've built evals, benchmarks, verifiers, reward models, RL environments, or synthetic or human data pipelines for LLMs or agents, with your own hands
You have a researcher's skepticism about whether a number means anything, and an engineer's instinct to ship code to find out
You've chosen risk before: founder, founding engineer, early employee, or a small, high-output team
You typically have 2–6 years of experience
Non-code domains like legal, finance, healthcare, or enterprise operations
Post-training (SFT, RLHF, RLVR)
Statistics for evaluation, benchmark design, or published eval research
Eval dashboards or observability tooling without hands-on eval design
Model training with little interest in evaluation
Founding equity and competitive cash
One of the first hires, working directly with the founder and customers from day one
A say in how agent quality gets measured across the industry
A beautiful office in FiDi
Breakfast, lunch, and dinner
Medical, dental, and vision coverage
Equinox membership or a fitness stipend
Essentially unlimited budget for AI tools and tokens
Relocation support if you're moving to San Francisco
Job details are sourced from the employer's original posting.
Open job postingAbout the company
Litmus provides software transforming machine data into operational insights for Industrial IoT, Industry 4.0, and Edge Computing.