About us:
Calder is solving the data bottleneck for embodied AI and world models. Today's AI models lack the real-world, 4D data needed to reason on spatial problems and operate in physical environments. We capture, process, and deliver high-fidelity human-generated spatial data at scale, providing the substrate that trains the next generation of generalizable models for manipulation, navigation, and interaction.
We're looking for a Senior Data Infrastructure Engineer to help build and run the petabyte-scale data platform behind Calder's spatial data products. Every capture — robot rigs, egocentric headsets, multi-camera video, depth, IMU — and everything derived from it — reconstructions, Gaussian Splats, labels, training datasets — lands in this platform, gets processed and versioned, and is served to customers and training clusters. You'll work on storage layout, table formats, versioning, ingest, and the distributed processing that turns raw sensor data into training-ready datasets, and help the platform hold up as volume grows by orders of magnitude. This is a hands-on role: you'll write, deploy, and operate core parts of the platform alongside the team. We're looking for someone who has personally built and run a PB-level platform in production, not only someone who has used one.
Build and scale Calder's PB-scale data lake / lakehouse for multimodal spatial data — object storage layout, partitioning, open table formats (Iceberg or equivalent), catalogs, and metadata
Work on data versioning and lineage (LakeFS or equivalent): branches, commits, reproducible dataset snapshots, and provenance from raw capture to customer delivery
Build high-throughput, fault-tolerant ingest for field and lab captures — upload, validation, checksumming, content-addressed deduplication, and safe promotion through bronze → silver → gold
Build and operate large-scale distributed processing across CPU and GPU fleets — video decode and frame extraction, sync and calibration application, SLAM / reconstruction / 3DGS batch jobs, and model-based auto-labeling (pose, depth, detection)
Build and maintain data pipelines at scale: idempotent jobs, retries, backfills, partitioned reprocessing, and clean recovery when a multi-day run fails halfway
Build the dataset curation and query layer — metadata indexing, SQL over the lake (Trino / DuckDB / Spark SQL), and search so researchers and customers can slice data without copying it
Keep GPUs fed: sharded training formats (WebDataset, Mosaic Streaming, or similar), caching, prefetching, and efficient cross-region movement between capture sites and training regions
Work on platform reliability, observability, and cost on AWS — S3 tiering and lifecycle, egress, spot and right-sizing, capacity planning, and alerting
Contribute to infrastructure-as-code (Terraform) and CI/CD for the data platform; write design docs and runbooks so others can operate it without tribal knowledge
Work with capture, calibration, SLAM, 3DGS, labeling, and QA teams to turn their workload requirements into platform capabilities
B.S. or M.S. in Computer Science, Software Engineering, or a related field, or equivalent practical experience
6+ years in data infrastructure, data platform, or distributed systems engineering
Hands-on experience setting up and operating a data platform at petabyte scale in production — you can speak concretely to the volumes, throughput, object counts, and failure modes you personally handled
Deep object storage experience (S3 or equivalent): key/prefix design, request-rate and throughput limits, the small-file problem, lifecycle and storage-class tiering, and cost control
Production experience with open table formats and lakehouse components — Iceberg, Delta Lake, or Hudi; catalogs (Glue, Polaris, Nessie, Unity, or similar); compaction and snapshot management
Strong experience with distributed processing — Spark, Ray, Flink, or Dask — tuned for large production jobs, not only used through notebooks
Experience with workflow orchestration (Airflow, Argo, Prefect, or similar) and the discipline that goes with it: idempotency, retries, backfills, and data contracts
Strong programming in Python plus at least one of Go, Rust, C++, Java, or Scala
Solid AWS infrastructure skills — EC2, EKS/ECS, S3, IAM, VPC, networking — and Terraform or equivalent IaC
Production operations experience: you've been on call for a data platform and care about its uptime, correctness, and cost
Clear written communication: design docs and runbooks other engineers can follow
Multimodal or unstructured data at scale — video, images, LiDAR / point clouds, sensor logs, or 3D assets — ideally from autonomous driving, robotics, mapping, or simulation
Data versioning systems (LakeFS, DVC, Nessie) and content-addressed storage / deduplication design
Feeding large-scale ML training: WebDataset, Mosaic Streaming, Petastorm, Ray Data, or custom loaders for multi-node GPU training
GPU-accelerated data processing (NVIDIA DALI, RAPIDS, NVDEC / FFmpeg pipelines)
Query engines over lake data (Trino, DuckDB, ClickHouse) and embedding / metadata search for dataset curation
Multi-region or hybrid cloud / on-prem architectures, including large-scale data transfer and egress optimization
Observability stacks (Datadog, Prometheus / Grafana, OpenTelemetry)
Robotics log formats (ROS bags, MCAP) and time-synchronized multi-sensor data
Contributions to open-source data infrastructure projects
Calder is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status. Learn more here.
By applying for this position, you consent to the collection, storage, and processing of your personal data by Calder for recruitment purposes. We will handle your information securely and in accordance with local privacy policies. Your data may be shared with authorized third-party service providers aiding in the hiring process.
Job details are sourced from the employer's original posting.
Open job postingAbout the company
A company operating through Ashby's recruitment platform.