HireFT
Browse JobsHow it worksPricingSuccess Stories
    Back to jobs
    TM

    Tms Llc

    IT Services

    Hiring | AI Engineer – Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote

    Los Angeles, United StatesRemoteContractPosted 1w ago
    All Tms Llc jobs

    Job description

    Role: AI Engineer – Agent Evaluation Specialist

    Location: Los Angeles, CA (Hybrid)

    Duration: 3+ Months (Possible Extension)

    Primary Focus: Hands-on Agent Evaluation

    Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation

    We are looking for an AI Engineer with strong, practical experience in evaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.

    Key Requirements (Must-Have):

    • LLM-as-a-Judge: Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
    • Golden Dataset Development: Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
    • Langfuse: Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
    • Agent Evaluation: Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.

    Critical Screening Criteria:

    Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.

    Agent evaluation is the core requirement and will be the primary selection criterion.

    All your information will be kept confidential according to EEO guidelines.

    Job details are sourced from the employer's original posting.

    Open job posting
    TM

    About the company

    Tms Llc

    TMS LLC

    View all Tms Llc jobs
    Industry
    IT Services
    Open roles
    12

    Interested in this role?

    Apply with HireFT

    Free to start — no card required.

    Your fit

    How well do you match?

    Sign in to see how your résumé lines up with this role.