About the Role
We are building AI systems that advisors and their clients rely on inside a regulated wealth management platform — an internal AI assistant, an external-facing API layer on top of it, and a growing set of agentic workflows. The bar for correctness, grounding, and auditability is higher here than it is for most AI products.
This role exists because traditional QA does not cover that surface. We need an engineer who can evaluate non-deterministic systems, build the tooling that catches AI regressions before they reach an advisor, and hold the line on code quality now that a meaningful share of our code is written by autonomous coding agents. This is a hands-on engineering role, not a test-execution role.
What You'll Do
1. Own and extend our AI evaluation framework for AI chat
- Build and maintain evaluation suite for our AI assistant and its API surface — correctness, grounding/citation fidelity, retrieval quality, refusal and escalation behavior, tone, latency, and cost per interaction.
- Curate and version golden datasets and adversarial test sets drawn from real advisor workflows; keep them representative as the product changes.
- Design LLM-as-judge and rubric-based scoring where deterministic assertions do not apply and validate the judges themselves against human-labeled sets.
- Wire evals into CI so prompt, model, retrieval, and tool changes are gated by measured regression — not by vibes.
- Instrument production traces and close the loop from live failures back into the eval suite.
2. Build the multi-agent AI quality step that finds defects automatically
- Design and operate a multi-agent workflow in our CI/CD pipeline that runs on commit: executes the suite, triages failures, writes up defects with reproduction detail, proposes or applies fixes, and re-runs to verify.
- Define agent roles, hand-off logic, guardrails, and the quality checks that decide when an agent's output is trustworthy enough to auto-apply versus route to a human.
- Make cost-aware calls on where an LLM us needed and where deterministic automation is simply the better tool.
- Report on the loop's real impact — autonomous resolution rate, false-positive rate, engineer hours returned.
3. Scale code quality in the age of autonomous agent coding
- Define what “good” means for agent-authored code and build the automated gates that enforce it: coverage and mutation testing, static analysis, security and dependency scanning, architectural conformance, and review checklists tuned to how agents fail.
- Identify the failure modes specific to agent-generated code — plausible-but-wrong logic, silent scope creep, duplicated abstractions, missing edge-case handling, tests written to pass rather than to verify — and build detection for them.
- Partner with architects and delivery teams to keep velocity from outrunning quality as agent-assisted development scales across the org.
Across all three
- Partner with product, engineering, and AI Labs to define coverage and acceptance criteria for AI features before they are built.
- Contribute to release-readiness and go/no-go decisions with evidence.
- Maintain and modernize our existing automation estate (API, UI, integration) and mentor engineers on
