HireFT
Browse JobsHow it worksPricingSuccess Stories
    Back to jobs
    LO

    Lockton

    Insurance Brokerage

    AI Quality Analyst, AI & Innovation

    Kansas City, Macao SAR ChinaOn-SiteFull-time2–5 yrs experiencePosted 4d ago
    All Lockton jobs

    Job description

    The AI Quality Analyst, AI & Innovation safeguards the quality, reliability, and measurable performance of the team’s AI data extraction processes, which convert commercial risk insurance documents — such as policies, endorsements, submissions, schedules of values, and loss runs — into structured data. Rather than building the extraction systems, this role exists to support the organization’s Enterprise AI Evaluation Strategies — the standards, methods, and processes used to determine whether AI systems are accurate, reliable, and ready for use. The role spans both the foundation and the practice of evaluation: building and curating the ground-truth (“golden”) datasets against which AI performance is measured, and contributing directly to the evaluation processes those datasets power. Reporting to the Director, AI & Innovation, the AI Quality Analyst creates, curates, and maintains ground-truth files, helps design and run evaluations, reviews AI extraction outputs against ground truth, and delivers clear, data-driven feedback that the development team uses to advance the extraction process.

    A central part of the role is engaging directly with business users: training them on the ground-truth application, guiding them through the process of producing and reviewing ground-truth data, and collecting the associated source files and validated outputs. The AI Quality Analyst maintains a comprehensive, working understanding of the extraction processes to define what “correct” looks like and to safeguard extraction quality end to end. The role is well suited to someone who brings a working knowledge of commercial risk insurance, is familiar with AI/ML and large language model concepts, and is skilled at quality review, data stewardship, and stakeholder engagement — without requiring hands-on AI/ML development experience.

    Key Responsibilities
    •    Own, create, and maintain the curated ground-truth (“golden”) datasets and ground-truth labels that developers rely on to measure, benchmark, and improve AI extraction quality — the foundation of the team’s evaluation work.
    •    Contribute to the design, execution, and continuous improvement of the organization’s Enterprise AI Evaluation Strategies, making sure ground-truth data is built to serve the evaluations it supports rather than simply collected and stored.
    •    Help design evaluation plans and test sets for AI extraction use cases, selecting representative and edge-case samples across commercial risk insurance document types, lines of business, and levels of complexity.
    •    Participate in evaluation cycles — baseline, regression, and pre-release testing — whenever models, prompts, or processes change, and help define the acceptance thresholds that determine whether changes are ready for production.
    •    Help calibrate automated evaluation methods, such as LLM-as-judge scoring, against human-reviewed ground truth to confirm those methods can be trusted at scale.
    •    Contribute to enterprise evaluation standards, templates, and playbooks so evaluation practices are consistent and reusable across AI initiatives beyond data extraction.
    •    Define, document, and uphold clear standards and annotation guidelines for what constitutes a correct extraction, so quality can be measured consistently and reproducibly.
    •    Apply commercial risk insurance knowledge to define what a correct extraction looks like for key policy and risk data — such as named insureds, lines of coverage, limits, deductibles and retentions, premiums, exclusions, and loss history — and to resolve ambiguous or complex cases.
    •    Engage with and train business users on the ground-truth application, guide them through creating and reviewing ground-truth data, and collect the associated source files and validated outputs needed to build and expand the ground-truth library.
    •    Review AI extraction outputs against ground truth, identify and categorize discrepancies and error types, and deliver actionable diagnostics and feedback to the team advancing the extraction process.
    •    Coordinate human-in-the-loop review and annotation workflows, including inter-annotator agreement, to keep ground-truth data accurate, consistent, and trustworthy.
    •    Manage the versioning and lifecycle of ground-truth datasets so quality results remain reproducible and comparable as models, prompts, and processes change.
    •    Partner with developers to define and interpret quality metrics and acceptance criteria — accuracy, precision, recall, F1, completeness, groundedness/faithfulness, hallucination rate, and schema conformance — and help translate results into clear findings.
    •    Monitor quality trends across extraction runs to surface drift, degradation, and anomalies before they reach downstream products, escalating issues to the team.
    •    Maintain a comprehensive, working understanding of the existing AI data extraction processes the team leads, partnering closely with the Manager and team in support of their continuous improvement.
    •    Support responsible-AI review of AI outputs, including robustness, bias, safety, and compliance considerations at the output level.
    •    Define and maintain quality metrics, dashboards, and reporting that give the team and stakeholders clear visibility into extraction performance and ground-truth coverage.
    •    Document ground-truth methodology, metric definitions, annotation standards, and quality practices to establish and mature the team’s AI quality practice.
    •    Collaborate with the Analytics Engineering function, which consumes validated AI-extracted data into Gold-layer assets, and the Applied AI Analyst, which evaluates AI from a business and product perspective, to align quality standards with downstream needs.

    The AI Quality Analyst, AI & Innovation safeguards the quality, reliability, and measurable performance of the team’s AI data extraction processes, which convert commercial risk insurance documents — such as policies, endorsements, submissions, schedules of values, and loss runs — into structured data. Rather than building the extraction systems, this role exists to support the organization’s Enterprise AI Evaluation Strategies — the standards, methods, and processes used to determine whether AI systems are accurate, reliable, and ready for use. The role spans both the foundation and the practice of evaluation: building and curating the ground-truth (“golden”) datasets against which AI performance is measured, and contributing directly to the evaluation processes those datasets power. Reporting to the Director, AI & Innovation, the AI Quality Analyst creates, curates, and maintains ground-truth files, helps design and run evaluations, reviews AI extraction outputs against ground truth, and delivers clear, data-driven feedback that the development team uses to advance the extraction process.

    A central part of the role is engaging directly with business users: training them on the ground-truth application, guiding them through the process of producing and reviewing ground-truth data, and collecting the associated source files and validated outputs. The AI Quality Analyst maintains a comprehensive, working understanding of the extraction processes to define what “correct” looks like and to safeguard extraction quality end to end. The role is well suited to someone who brings a working knowledge of commercial risk insurance, is familiar with AI/ML and large language model concepts, and is skilled at quality review, data stewardship, and stakeholder engagement — without requiring hands-on AI/ML development experience.

    Key Responsibilities
    •    Own, create, and maintain the curated ground-truth (“golden”) datasets and ground-truth labels that developers rely on to measure, benchmark, and improve AI extraction quality — the foundation of the team’s evaluation work.
    •    Contribute to the design, execution, and continuous improvement of the organization’s Enterprise AI Evaluation Strategies, making sure ground-truth data is built to serve the evaluations it supports rather than simply collected and stored.
    •    Help design evaluation plans and test sets for AI extraction use cases, selecting representative and edge-case samples across commercial risk insurance document types, lines of business, and levels of complexity.
    •    Participate in evaluation cycles — baseline, regression, and pre-release testing — whenever models, prompts, or processes change, and help define the acceptance thresholds that determine whether changes are ready for production.
    •    Help calibrate automated evaluation methods, such as LLM-as-judge scoring, against human-reviewed ground truth to confirm those methods can be trusted at scale.
    •    Contribute to enterprise evaluation standards, templates, and playbooks so evaluation practices are consistent and reusable across AI initiatives beyond data extraction.
    •    Define, document, and uphold clear standards and annotation guidelines for what constitutes a correct extraction, so quality can be measured consistently and reproducibly.
    •    Apply commercial risk insurance knowledge to define what a correct extraction looks like for key policy and risk data — such as named insureds, lines of coverage, limits, deductibles and retentions, premiums, exclusions, and loss history — and to resolve ambiguous or complex cases.
    •    Engage with and train business users on the ground-truth application, guide them through creating and reviewing ground-truth data, and collect the associated source files and validated outputs needed to build and expand the ground-truth library.
    •    Review AI extraction outputs against ground truth, identify and categorize discrepancies and error types, and deliver actionable diagnostics and feedback to the team advancing the extraction process.
    •    Coordinate human-in-the-loop review and annotation workflows, including inter-annotator agreement, to keep ground-truth data accurate, consistent, and trustworthy.
    •    Manage the versioning and lifecycle of ground-truth datasets so quality results remain reproducible and comparable as models, prompts, and processes change.
    •    Partner with developers to define and interpret quality metrics and acceptance criteria — accuracy, precision, recall, F1, completeness, groundedness/faithfulness, hallucination rate, and schema conformance — and help translate results into clear findings.
    •    Monitor quality trends across extraction runs to surface drift, degradation, and anomalies before they reach downstream products, escalating issues to the team.
    •    Maintain a comprehensive, working understanding of the existing AI data extraction processes the team leads, partnering closely with the Manager and team in support of their continuous improvement.
    •    Support responsible-AI review of AI outputs, including robustness, bias, safety, and compliance considerations at the output level.
    •    Define and maintain quality metrics, dashboards, and reporting that give the team and stakeholders clear visibility into extraction performance and ground-truth coverage.
    •    Document ground-truth methodology, metric definitions, annotation standards, and quality practices to establish and mature the team’s AI quality practice.
    •    Collaborate with the Analytics Engineering function, which consumes validated AI-extracted data into Gold-layer assets, and the Applied AI Analyst, which evaluates AI from a business and product perspective, to align quality standards with downstream needs.

    •    2–5+ years of experience in data quality, quality assurance, data analysis, annotation/labeling, or a related analytical or business-facing role, ideally with exposure to AI/ML, LLM, or data-extraction systems. Hands-on AI/ML or software development experience is not required.
    •    Experience contributing to structured testing, validation, or evaluation processes, such as QA test plans, user acceptance testing, or data or model validation.
    •    Working knowledge of commercial risk insurance — including policy structure, coverage terms, limits, deductibles, exclusions, endorsements, submissions, and loss runs — gained through underwriting, brokerage, claims, risk management, or insurance operations, sufficient to judge whether extracted data is accurate and complete.
    •    General familiarity with large language models and generative AI — what they do well, where they fall short, and why they can produce inaccurate or fabricated outputs — sufficient to understand, review, and reason about AI outputs.
    •    Strong analytical rigor and exceptional attention to detail, with the ability to assess extraction accuracy and translate findings into clear, actionable recommendations.
    •    Excellent communication and stakeholder-engagement skills, with the ability to train and support business users and to build effective ground-truth workflows with non-technical partners.
    •    Comfort working with data and data tools.

    Preferred Qualifications
    •    Bachelor’s degree in Computer Science, Information Systems, Mathematics, or a related field.
    •    Familiarity with technical generative-AI concepts such as prompt engineering, retrieval-augmented generation (RAG), and LLM-as-judge evaluation.
    •    Understanding of evaluation methodology and metrics for AI/ML and information extraction, such as precision, recall, F1 score, accuracy, and hallucination detection.
    •    Experience building, curating, or maintaining labeled or ground-truth datasets and annotation workflows, including inter-annotator agreement.
    •    Insurance industry designations such as CPCU, AINS, or ARM.
    •    Basic proficiency in SQL for querying evaluation and production data.
    •    Basic scripting or Python familiarity; the role is centered on quality review and data stewardship rather than software development.
    •    Familiarity with AI/LLM observability and evaluation tooling such as LangSmith.
    •    Familiarity with the Databricks platform and tools such as MLflow and Unity Catalog.

    Success Measures
    •    Coverage, accuracy, and freshness of the ground-truth datasets, and how well they support the evaluations they are built for.
    •    Meaningful contribution to Enterprise AI Evaluation Strategies, including reliable evaluation cycles, clear acceptance criteria, and adoption of evaluation standards across AI initiatives.
    •    The degree to which developers can quickly and confidently measure and validate extraction quality using the ground-truth files provided.
    •    Adoption and proficiency of business users on the ground-truth application, and the volume and quality of source files and validated outputs collected.
    •    Early detection of quality drift, degradation, and anomalies before they impact downstream products.
    •    Clarity and usefulness of the quality metrics, dashboards, diagnostics, and feedback provided to the team and stakeholders.

    #LI-JM

    •    2–5+ years of experience in data quality, quality assurance, data analysis, annotation/labeling, or a related analytical or business-facing role, ideally with exposure to AI/ML, LLM, or data-extraction systems. Hands-on AI/ML or software development experience is not required.
    •    Experience contributing to structured testing, validation, or evaluation processes, such as QA test plans, user acceptance testing, or data or model validation.
    •    Working knowledge of commercial risk insurance — including policy structure, coverage terms, limits, deductibles, exclusions, endorsements, submissions, and loss runs — gained through underwriting, brokerage, claims, risk management, or insurance operations, sufficient to judge whether extracted data is accurate and complete.
    •    General familiarity with large language models and generative AI — what they do well, where they fall short, and why they can produce inaccurate or fabricated outputs — sufficient to understand, review, and reason about AI outputs.
    •    Strong analytical rigor and exceptional attention to detail, with the ability to assess extraction accuracy and translate findings into clear, actionable recommendations.
    •    Excellent communication and stakeholder-engagement skills, with the ability to train and support business users and to build effective ground-truth workflows with non-technical partners.
    •    Comfort working with data and data tools.

    Preferred Qualifications
    •    Bachelor’s degree in Computer Science, Information Systems, Mathematics, or a related field.
    •    Familiarity with technical generative-AI concepts such as prompt engineering, retrieval-augmented generation (RAG), and LLM-as-judge evaluation.
    •    Understanding of evaluation methodology and metrics for AI/ML and information extraction, such as precision, recall, F1 score, accuracy, and hallucination detection.
    •    Experience building, curating, or maintaining labeled or ground-truth datasets and annotation workflows, including inter-annotator agreement.
    •    Insurance industry designations such as CPCU, AINS, or ARM.
    •    Basic proficiency in SQL for querying evaluation and production data.
    •    Basic scripting or Python familiarity; the role is centered on quality review and data stewardship rather than software development.
    •    Familiarity with AI/LLM observability and evaluation tooling such as LangSmith.
    •    Familiarity with the Databricks platform and tools such as MLflow and Unity Catalog.

    Success Measures
    •    Coverage, accuracy, and freshness of the ground-truth datasets, and how well they support the evaluations they are built for.
    •    Meaningful contribution to Enterprise AI Evaluation Strategies, including reliable evaluation cycles, clear acceptance criteria, and adoption of evaluation standards across AI initiatives.
    •    The degree to which developers can quickly and confidently measure and validate extraction quality using the ground-truth files provided.
    •    Adoption and proficiency of business users on the ground-truth application, and the volume and quality of source files and validated outputs collected.
    •    Early detection of quality drift, degradation, and anomalies before they impact downstream products.
    •    Clarity and usefulness of the quality metrics, dashboards, diagnostics, and feedback provided to the team and stakeholders.

    #LI-JM

    Job details are sourced from the employer's original posting.

    Open job posting
    LO

    About the company

    Lockton

    A privately held insurance brokerage and risk management consulting firm providing commercial insurance, employee benefits, and related services.

    View all Lockton jobs
    Industry
    Insurance Brokerage
    Open roles
    251

    Interested in this role?

    Apply with HireFT

    Free to start — no card required.

    Your fit

    How well do you match?

    Sign in to see how your résumé lines up with this role.