Job Title:Data Engineer - ML Training Data Pipeline Notice period: 0-30 Days
Experience:
5+ Years
Location:
Hyderabad OR Pune We are looking for Data Engineer - ML Training Data Pipeline who can Build and maintain the data pipeline that transforms raw production traces into high-quality training datasets for LLM fine-tuning-ingestion, deduplication, format conversion, quality filtering, and train/test splitting at scale on AWS. What We Expect: Build end-to-end data pipelines: raw trace ingestion → dedup → format conversion → quality gating → training-ready datasets Process large-scale JSONL data on AWS S3 (tens of thousands of traces per batch) Convert between chat-completion formats (e.g., OpenAI → Llama 3.1 tool-calling format) Implement smart deduplication and sampling to balance training distribution Design identity-aware train/test splits that measure true generalization Build data validation gates to detect schema drift and format anomalies Create a continuous pipeline that auto-processes new production traces for retraining Requirements
Experience:
Benefits:
Job details are sourced from the employer's original posting.
Open job postingAbout the company
DataEconomy provides data management and analytics solutions for businesses.