Job Description
As a Lead Data Engineer, you will be responsible for overseeing the design, development, implementation, and maintenance of the data infrastructure and pipelines within the organization. You will lead a team of data engineers and collaborate with cross-functional teams to ensure the efficient and effective management of data assets. This role requires a strong understanding of data engineering principles, excellent technical skills, and the ability to provide technical leadership.
Responsibilities:
- Lead and manage a team of data engineers, providing guidance, mentoring, and support to ensure successful project delivery.
- Collaborate with stakeholders, including data scientists, analysts, and product managers, to understand data requirements and translate them into technical solutions.
- Design and develop scalable, reliable, and efficient data pipelines and ETL processes to extract, transform, and load data from various sources into data warehouses or data lakes.
- Implement and maintain data governance and data quality frameworks to ensure data accuracy, integrity, and compliance with privacy regulations.
- Optimize data infrastructure and architecture to improve data processing performance and reduce latency.
- Identify and evaluate new technologies, tools, and frameworks to enhance data engineering capabilities and drive innovation.
- Collaborate with infrastructure and operations teams to ensure the availability, reliability, and security of data platforms.
- Define and enforce coding standards, best practices, and development methodologies to ensure high-quality code and maintainable solutions.
- Conduct performance reviews, provide feedback, and support the professional growth and development of team members.
- Stay up-to-date with industry trends, emerging technologies, and advancements in data engineering to drive continuous improvement and innovation within the team.
Qualifications:
- Bachelor's or Master's degree in Computer Science, Information Technology or any other field, preferably from a premium/Tier-1 college or institute/university.
- Proven experience in data engineering, with a focus on designing and implementing data pipelines, ETL processes, and data warehousing solutions.
- Good experience in Azure Data Factory, Data Lake, Azure Synapse,Databricks, building data models/data marts to be used in Power BI
- Skilled in end-to-end ETL/ELT process. with Bigdata eco systems
- Strong Hands-on experience in writing efficient code in Python/Pyspark to manipulate data and draw insights and have used libraries such as - Pandas, Numpy, Matplotlib, modin, Dask, scikit-learn etc.
- A successful history of manipulating, processing and extracting value from large disconnected datasets, including MariaDB, Mongo, Elasticsearch index.
- Familiar with data extraction using third party APIs
- Proficiency in working with relational databases (SQL) and NoSQL databases.
- Excellent problem-solving and analytical skills with the ability to translate complex business requirements into technical solutions.
