As a Senior Data Engineer on the Financial Flows team, you will take full technical ownership of the end-to-end data pipelines, predictive models, and cloud infrastructure powering Kpler’s commodity finance analytics . You will lead the modernization of our data foundation, spearheading the migration of production services to AWS and re-architecting legacy calculation engines into clean, performant Python services . Operating at the crossroads of quantitative research and cloud infrastructure, your work will directly ensure that market participants receive real-time, mission-critical insights across global commodities markets .
Design, build, and operate resilient batch and real-time streaming data ingestion pipelines connecting to external financial market data providers via APIs and WebSockets .
Lead the architectural migration of legacy backend infrastructure to modern cloud-native environments on AWS, establishing high standards for system reliability and scalability .
Drive the refactoring and rewrite of the CTA positioning indicator calculation blocks from PHP to Python, ensuring strict mathematical output parity, comprehensive test coverage, and optimized latency .
Ingest and model complex financial datasets—including futures pricing, open interest, and trading volumes—across time-series stores and search engines like QuestDB and Elasticsearch .
Implement automated data validation, proactive alerting, and system monitoring to ensure market data feeds remain accurate, continuous, and fault-tolerant .
Deploy, maintain, and scale containerized services across Docker and Kubernetes environments using automated CI/CD pipelines and GitOps workflows .
Partner closely with quantitative researchers, commodity analysts, and product managers to translate complex trading models into high-performance production pipelines .
Provide technical mentorship to peer engineers, raising team-wide engineering rigor through thoughtful code reviews, system design guidance, and architectural documentation .
5+ years of professional experience building, operating, and scaling production-grade data pipelines and backend infrastructure .
Advanced expertise in Python, with a strong background refactoring and rewriting production-critical code for performance and testability .
Proven hands-on experience building and maintaining real-time or streaming API ingestion pipelines (REST, WebSockets) handling time-series datasets .
Solid production experience deploying and managing containerized applications on AWS using Docker, Kubernetes, and modern CI/CD practices .
Strong database fundamentals in SQL and experience working with distributed search or time-series databases (e.g., Elasticsearch, QuestDB, ClickHouse) .
Deep experience establishing automated data quality frameworks, observability (monitoring/alerting), and root-cause troubleshooting for production systems .
Domain knowledge of financial market structures, futures contracts, open interest, and commodities trading symbology .
Experience integrating data from major financial market vendors (e.g., Bloomberg, LSEG, Enverus, Trading Technologies) .
Familiarity with legacy PHP codebase migration or reading PHP algorithms to translate into Python .
Practical familiarity with GitOps frameworks and infrastructure-as-code tooling.
Job details are sourced from the employer's original posting.
Open job postingAbout the company
Kpler is a leading data and intelligence company specializing in the energy and commodities markets. They provide real-time analytics and insights to help businesses make informed decisions in the oil and gas sector.