Remote position - only for professionals based in LATAM.
We are looking for a Senior Data Engineer for one of our client's teams. You will take ownership of the systems responsible for ingesting, transforming, validating, and publishing data across our platform.
In this role, you will work at the data ingestion boundary, where data from multiple external sources enters our systems. You will be responsible for building reliable pipelines, resolving identity and entity-matching challenges, detecting data-quality issues, and ensuring data flows correctly into downstream products and services.
This is a hands-on engineering role that combines AWS data engineering, Python/SQL development, event-driven architectures, data quality, and identity resolution. You will collaborate closely with engineering, product, analytics, and downstream platform teams to troubleshoot issues and improve the reliability of our data ecosystem.
What You'll Do
• Own and evolve data ingestion pipelines, including source ingestion/scraping, data cleaning and curation using AWS Glue, entity resolution, and event-driven data publishing.
• Design and maintain reliable batch and event-driven data workflows across our data platform.
• Diagnose and resolve identity-resolution issues across multiple data sources, including deduplication and entity-matching challenges.
• Develop and maintain data quality and validation frameworks, including freshness monitoring, null-rate checks, consistency validation, and schema-drift detection.
• Monitor data at the ingestion boundary and proactively identify issues before they impact downstream systems and products.
• Partner with engineering teams responsible for the events bridge and downstream identity services to trace data and events end-to-end.
• Collaborate with Product, Assessments, Analytics, and other stakeholders to understand how data flows into downstream products and ensure those requirements are reflected in the data pipelines.
• Automate infrastructure and pipeline changes using Infrastructure as Code, primarily Terraform or AWS CDK.
• Work with AWS services including Glue, Athena, S3, and event-driven services to build and operate scalable data infrastructure.
• Participate in production incident response, quickly identifying the scope and impact of data-quality issues and implementing remediation.
• Continuously improve pipeline reliability, observability, maintainability, and operational processes.
What You'll Bring
• 5+ years of experience building and operating production-grade data pipelines.
• Strong experience working with both batch data processing and event-driven architectures.
• Hands-on experience with AWS Glue, Athena, and S3-based data lakes, including layered/medallion-style data transformations.
• Strong proficiency in Python and SQL for data transformation, processing, and analysis.
• Experience with identity resolution, entity matching, or deduplication, including exact and fuzzy matching approaches.
• Understanding of the challenges and tradeoffs involved in maintaining durable identifiers and first-seen/locked identity mappings.
• Experience working with event schemas and schema-registry-backed contracts, such as Protobuf, and an understanding of the risks associated with schema changes.
• Strong troubleshooting and production incident-response skills, including the ability to determine impact, identify root causes, and implement fixes quickly.
• Ability to understand and debug code written in a functional or concurrent programming language, such as Elixir.
• Strong communication and collaboration skills, with the ability to work effectively across engineering, product, analytics, and other technical teams.
Nice to Have
• Experience with Elixir/Phoenix or another BEAM-based concurrent processing framework.
• Experience with DynamoDB-backed identity, lookup, or matching services.
• Experience with the Snowflake ecosystem, including data modeling, Snowpipe, Streams, and Tasks.
• Experience working with
sports data providers, such as
MLB.com or Sportradar, or with other licensed-content/data provider ecosystems.
• Experience implementing data observability, including freshness/staleness alerts, null-rate monitoring, schema-drift detection, and data-quality dashboards.
• Experience working with Infrastructure as Code using Terraform or AWS CDK.
Technologies
Languages: Python, SQL, familiarity with Elixir
AWS: Glue, Athena, S3, DynamoDB
Data & Streaming: Data Lakes, Event-Driven Architecture, Protobuf, Schema Registries
Infrastructure: Terraform, AWS CDK
Data Platforms: Snowflake
Engineering Practices: Data Quality, Data Observability, Identity Resolution, Entity Matching, Incident Response
