Job Summary:
We are seeking a skilled Data Engineer with 3-5 years of experience to join our dynamic team. The ideal candidate will be responsible for developing, maintaining, and optimizing our data infrastructure and pipelines. You will work closely with data scientists, analysts, and other stakeholders to ensure data is accessible, reliable, and efficiently processed.
Key Responsibilities:
- Develop and Maintain Data Pipelines: Design, build, and manage scalable data pipelines for data extraction, transformation, and loading (ETL).
- Data Integration: Integrate data from various sources, ensuring data quality and consistency.
- Optimize Data Systems: Identify and implement process improvements to enhance data reliability, efficiency, and quality.
- Collaborate with Teams: Work with data scientists, analysts, and other stakeholders to support their data infrastructure needs.
- Monitor and Troubleshoot: Monitor data pipelines and systems, troubleshoot issues, and ensure data integrity.
- Documentation: Maintain comprehensive documentation of data processes and systems.
Required Skills and Qualifications:
- Experience: 3-5 years of experience in data engineering or a related field.
- Technical Skills: Proficiency in Py-Spark (Must Have), SQL, Python, and data platform like Databricks.
- Storage Management (Must Have): Experience in storage mechanism in AWS S3, Layers of Medallion Architecture, OTF like Delta, iceberg etc., Hand-Ons in Data schema like Star, snowflake etc.
- Problem-Solving: Strong analytical and problem-solving skills.
- Communication: Excellent communication and teamwork skills.
Preferred Qualifications:
- Cloud Platforms: Experience with cloud platforms (Mandatory: Databricks, Good To have: AWS tools like redshift , Glue , Airflow etc.).
- Big Data Technologies: Familiarity with big data technologies (e.g., Hadoop, Spark).
- Agile Methodologies: Experience working in Agile environments.
