HireFT
Browse JobsHow it worksPricingSuccess Stories
    Back to jobs
    CR

    Crux AI

    Artificial Intelligence

    Principal AI Ops Engineer

    Palo Alto, United StatesOn-SiteFull-time10+ yrs experiencePosted 1w ago
    All Crux AI jobs

    Job description

    Built to set the gold standard for integrated AI infrastructure

    Crux AI is a newly formed, U.S.-based integrated AI infrastructure company created to remove the physical and operational constraints on consequential AI ambitions. Crux brings together power, high-density data centers, TPU silicon, networking, orchestration software, and ongoing operations as one integrated system.

    Crux is being capitalized to plan every layer together, develop each one to demanding standards, and operate the whole system with efficiency and reliability. That gives hyperscalers, frontier AI labs, sovereign customers, enterprises, and AI-native companies greater freedom to pursue the AI they are here to create.

    WHAT YOU'LL DO

    You will build the systems that let AI agents, not people, do most of the work of running Crux's TPU fleet. Reporting to our Director, AI Ops, you will be one of the first engineers on the team, writing the telemetry pipelines, correlation logic, and agents that detect, triage, and remediate failures across facilities, TPU hosts, the network, and the control plane before a customer notices. You will turn every runbook into code and every repeat incident into automation. This is a hands-on engineering seat: you write and ship the software, then operate it in production.

    In this role, you will:

    • Build and own the telemetry and event pipeline that ties together power, cooling, TPU hosts, interconnect, and the control plane, with the correlation to tell a failing pump from a failing chip from a failing job.

    • Write AI agents that triage alerts, execute runbooks, drain and requeue workloads, and open hardware repair and RMA requests with Google, with every action logged, reversible, and measured.

    • Turn runbooks into code, so that every critical failure mode for the first build maps to a signal, an alert, an automated response, and an owner.

    • Build the guardrails that keep autonomous actions safe: blast-radius limits, approval gates for high-risk changes, and tested rollback paths.

    • Define and instrument SLOs and error budgets with Compute, Network Engineering, and Facility Operations, and build the views the team reviews every week.

    • Take part in on-call and incident response, and drive postmortems that end in a fix that ships, not an action item that ages.

    • Measure the share of operational work that closes with zero human touch, and drive that number up release over release.

    SIGNALS OF SUCCESS

    After 60 days in this role:

    • Gained a complete understanding of the first-build architecture, the failure modes that matter most, and the tooling in place today, and shipped a first working agent against a real, repeat failure.

    • Stood up the first version of the unified telemetry pipeline for the first build.

    • Adapted to a fast-paced environment, triaging appropriately and continuously improving our function.

    After 6 months:

    • Taken the first Stage 1 cluster into production with automated detection and remediation in place for its top failure modes.

    • Shipped agents the team relies on every day, with a measured share of incidents resolved without human intervention.

    • Built the guardrails and audit trail that let the team trust autonomous actions in production.

    After 1 year:

    • Running automation across a multi-site fleet where most repeat operational work executes autonomously, with measurable gains in availability, recovery time, and cost per chip.

    • Built a library of agents and runbooks-as-code that every new site inherits on day one.

    • Become a reason strong SRE and operations engineers choose Crux.

    EXPERIENCES, ATTRIBUTES AND MINDSET THAT INDICATE A GOOD MATCH

    Experiences

    • 10+ years in software engineering, SRE, or production engineering for large-scale infrastructure, including operating a fleet across many sites against hard SLAs.

    • Built and shipped AI-driven operations systems in production, such as event correlation, anomaly detection, auto-remediation, or agentic triage, with measurable results.

    • Strong software skills in Python or Go, and hands-on experience with observability stacks (for example Prometheus or OpenTelemetry), Kubernetes, and large-scale data pipelines.

    Attributes

    • You are outcome driven and hands-on. You read the telemetry and write the agent yourself. This is a builder's seat, not a supervisory one.

    • You are close enough to the hardware to be useful. You understand how power, cooling, accelerators, and networks fail, not just how software does.

    • You have a high tolerance for ambiguity. You absorb ambiguity and navigate complexity rather than passing it along.

    Mindset

    • You are skeptical by default. "90% automated" is a hypothesis until it survives a bad night at a new site. You build the system that makes the next incident boring.

    • You deliver reliability through discipline, not heroics. You would rather fix the cause once than page someone twice.

    • You are AI-agentic first. Fluent with AI agents, or committed to becoming so quickly, you embed them as first principles in how you and your team work, defaulting to agentic workflows before adding headcount or process.

    Nice to have (Preferred, not required):

    • SRE or production engineering at a hyperscaler (Google, Meta, Microsoft, AWS) or a neocloud (CoreWeave, Nebius, Lambda, Crusoe).

    • Experience operating TPU or GPU clusters for large training and inference workloads, including hardware health, repair workflows, and fleet telemetry.

    • Experience with data center infrastructure telemetry such as BMS, DCIM, or power and cooling systems.

    • Experience building LLM-based agents with tool use, evaluations, and safety controls in production.

    • Experience with predictive maintenance, ML-based anomaly detection, or failure prediction at fleet scale.

    Salary Range Information

    The annual salary range for this position has been estimated based on market data and other factors. However, a salary higher or lower than this range may be appropriate for a candidate whose qualifications differ meaningfully from those listed in the job description.

    About Crux

    • We offer generous base, bonus and additional incentive based compensation

    • Health, dental, and vision coverage for you and your dependents

    • Company-paid life insurance and disability

    • Full suite of other optional benefits

    • 401(k) Plan with 4% company match

    • Hybrid schedule offering four days in-office collaboration paired with one remote workday for focused, individual work

    Job details are sourced from the employer's original posting.

    Open job posting
    CR

    About the company

    Crux AI

    Built to set the gold standard for integrated AI infrastructure   Crux AI is a newly formed, U.S.-based integrated AI infrastructure company created to remove the physical and operational constraints on the world’s most consequential AI ambitions. Crux will finance, develop, and operate the full chain required to deliver large-scale TPU compute—from power and high-density data centers to silicon, networking, orchestration software, and ongoing operations—as one integrated system.   Those ambitions extend far beyond frontier models. They include faster drug discovery, earlier diagnoses, personalized treatment, mathematical breakthroughs, adaptive education, robotics for disaster response, energy optimization, national security, and data sovereignty. Their progress depends on reliable access to large blocks of accelerated compute—and on the power, facilities, networks, software, and technical expertise required to keep that compute running.   Crux is being capitalized and designed to address these constraints together. Its integrated model supports long-range capacity planning, efficient data center development, optimized TPU clusters, flexible service commitments, and dependable production operations. Crux will serve hyperscalers, frontier AI labs, sovereign and public-sector customers, global enterprises, and AI-native companies through compute-as-a-service and related infrastructure services.   This is the standard Crux is being built to set: every layer planned together, financed for the long term, and operated as one system—giving customers the freedom to focus on the AI they are here to create.

    View all Crux AI jobs
    Industry
    Artificial Intelligence
    Open roles
    36

    Interested in this role?

    Apply with HireFT

    Free to start — no card required.

    Your fit

    How well do you match?

    Sign in to see how your résumé lines up with this role.