HireFT
Browse JobsHow it worksPricingSuccess Stories
    Back to jobs
    VA

    Vanilla Steel GmbH

    Steel

    Senior QA Engineer

    Berlin, GermanyOn-SiteFull-time5+ yrs experiencePosted 1w ago
    All Vanilla Steel GmbH jobs

    Job description

    Your mission

    A senior, hands-on engineering role responsible for creating measurable confidence in both the conventional software platform and the agentic AI workflows behind it. You'll test the complete journey from incoming emails and files through extraction, material matching, RFQ creation and quotation - verifying not only that the software executes correctly, but that the resulting business data and AI decisions are accurate, safe and commercially sensible. The objective is to turn quality from manual judgment into an automated, evidence-based release process. Your gates should answer: does the application work, are the integrations behaving, did the AI extract the right information, did the agent choose the right tool with valid arguments, is the business outcome correct, has this change regressed anything, and is the release safe to deploy.
    WHAT YOU'LL OWN
    • Test strategy. Define and own the testing strategy across the platform - what gets tested at which layer (unit, component, contract, integration, end-to-end, production verification), where deterministic assertions end and statistical evaluation begins, and how quality gates map to release risk. Keep it a living document engineers actually use, not a compliance artifact.
    • Software test automation. Automated coverage across the React web app, backend APIs and services, auth and authorization, database operations, third-party integrations, email and document ingestion, background jobs and async workflows, multi-tenant configuration and isolation, and end-to-end RFQ and quotation journeys. Playwright, pytest, API clients, SQL, CI/CD. Focus on critical business journeys and high-risk failure modes, not test-case count.
    • API and integration testing. Beyond happy-path responses: request and response schemas, permissions, invalid and incomplete inputs, error handling, idempotency and duplicate requests, timeouts and retries, async state transitions, webhooks and event processing, data consistency across services, tenant isolation, and integration failure and recovery - running in CI with enough diagnostic output to explain a failure.
    • AI and LLM evaluations. Versioned evaluation datasets built from real business scenarios, edge cases and observed failures. Targets include extraction accuracy, material and product matching, classification and routing, agent decision-making, generated RFQ and quotation content, grounding, hallucination frequency, instruction adherence and final business correctness. You define the scoring methods - exact assertions, semantic comparison, rule-based and model-based graders, human review where needed. Because behaviour is nondeterministic, expect quality thresholds and statistical comparison rather than identical output every run.
    • Agent workflow testing. Inspect the whole execution path, not the final response: was the correct tool selected, were its arguments valid, were calls made in the right order, was the right source data used, were permissions and tenant boundaries respected, did retries and recovery work, did the agent stop or escalate when uncertain, did it avoid unnecessary actions, did the outcome match the business objective, and were latency and cost acceptable.
    • Controlled use of AI testing agents. Use bounded agents to analyse requirements, suggest missing scenarios, generate candidate tests, explore unusual journeys, create adversarial inputs, reproduce defects and analyse traces. You stay responsible for validating what they generate and for defining authoritative pass/fail criteria - AI output never becomes an uncontrolled release decision.
    • Release readiness. Define and maintain the criteria: critical journeys pass, API contracts stable, no cross-tenant access violations, extraction accuracy above the agreed threshold, no material regression in agent task completion, known critical scenarios still succeeding, latency and cost within limits. You communicate what passed, what failed, what's uncertain and the risk of proceeding, and give a recommendation - final release ownership stays shared with engineering and product.
    • Failure investigation. Work a regression across layers - browser traces and screenshots, API requests, application logs, agent execution traces, database queries, background-job status, model outputs and grader results, deployment and config changes - and turn every meaningful production failure into a reproducible test or evaluation case.
    WHAT YOU'LL PRODUCE 
    Automated UI and API suites · integration and end-to-end scenarios · version-controlled AI evaluation datasets · agent-workflow graders · CI/CD quality gates · test and evaluation reports · release-readiness dashboards · regression and failure classifications · defect reproduction packages · quality-risk assessments · a living test strategy documenting layer ownership, coverage standards and evaluation methodology.

    Your profile

    • 5+ years in SDET, quality engineering, test automation or software engineering.
    • Strong TypeScript or Python, with experience designing maintainable test frameworks - not only writing test cases.
    • Strong Playwright, pytest or comparable automation experience, and advanced API and integration testing knowledge.
    • Experience defining test strategy for a platform or organisation - deciding what gets tested where and why, not only implementing someone else's plan.
    • Experience testing asynchronous and distributed systems.
    • SQL, logging and production-debugging skills; CI/CD and automated release gates.
    • Strong judgment when requirements or outcomes are ambiguous - and the ability to distinguish technical correctness from business correctness.
    • EU work permission.
    USEFUL ADDITIONAL EXPERIENCE 
    • LLM and agent evaluations, trace-based debugging, model-based grading, prompt-injection and adversarial testing.
    • Multi-tenant SaaS platforms; email and document processing.
    • Procurement, manufacturing or supply-chain systems; data-quality and matching problems.

    Job details are sourced from the employer's original posting.

    Open job posting
    VA

    About the company

    Vanilla Steel GmbH

    Vanilla Steel GmbH is a company operating in the steel industry.

    View all Vanilla Steel GmbH jobs
    Industry
    Steel
    Open roles
    4

    Interested in this role?

    Apply with HireFT

    Free to start — no card required.

    Your fit

    How well do you match?

    Sign in to see how your résumé lines up with this role.