As a Senior Data Engineer joining our new Marine Insurance initiative, you will play a central role in shaping the systems that power Kpler’s global market intelligence platform . You will lead the end-to-end design and delivery of complex data pipelines and cloud infrastructure, ingesting, processing, and serving large-scale data to our global customers . Acting as a technical leader, you will set data architecture directions and mentor peer engineers, ensuring our platform reliably scales as our datasets and product offerings expand .
Design and own end-to-end data pipelines: Build, deploy, and operate production-grade data pipelines and infrastructure from initial architecture through deployment, monitoring, and maintenance .
Deploy containerized AWS services: Develop and containerize applications within AWS using Python, Docker, Kubernetes, Airflow, and Infrastructure as Code (Terraform) .
Ensure data integrity and event streaming: Architect robust PostgreSQL database structures and event-streaming flows (Kafka/Flink) to guarantee seamless data flow and integrity .
Set technical direction and standards: Lead complex pipeline design, peer reviews, and logging, monitoring, and alerting frameworks to maintain pipeline resilience .
Partner cross-functionally and mentor peers: Collaborate closely with engineers, product managers, and data scientists while mentoring engineers to elevate team craft and technical development .
Evaluate emerging data tools: Research and implement modern data infrastructure improvements and frameworks to continuously elevate Kpler's platform capabilities .
Data engineering experience at scale: 5+ years of experience designing, operating, and owning production-grade data pipelines and infrastructure .
Core tech stack depth: Strong Python skills with hands-on experience in Docker, Kubernetes, AWS, Airflow, and relational databases like PostgreSQL .
Streaming, IaC, and observability: Practical experience with event streaming (Kafka/Flink), Infrastructure as Code (Terraform), CI/CD practices, and production logging/monitoring systems .
Technical leadership & AI coding tools: Demonstrated ability to lead architectural discussions, mentor engineers, and leverage AI-assisted coding tools (Copilot, Cursor, Claude) with strong technical judgment .
Methodologies & communication: Experience working in Agile environments with strong collaboration skills and fluency in written and spoken English .
Large-scale analytics platforms: Experience building large-scale data processing systems or analytics platforms .
Cross-team initiatives: Track record of leading technical initiatives across multiple teams .
Domain knowledge: Familiarity with commodities, shipping, energy, or marine insurance market data .
Job details are sourced from the employer's original posting.
Open job postingAbout the company
Kpler is a leading data and intelligence company specializing in the energy and commodities markets. They provide real-time analytics and insights to help businesses make informed decisions in the oil and gas sector.