- Design, develop, and maintain scalable data pipelines and systems.
- Work closely with data scientists and analysts to ensure data quality and availability.
- Implement data integration and transformation processes using PySpark and Python.
- Optimize and maintain SQL databases and queries.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Monitor and troubleshoot data pipeline issues to ensure data integrity and performance.
Required Skills and Qualifications:
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- 3+ years of experience in data engineering.
- Proficiency in PySpark, Python, and SQL.
- Experience with Azure Databricks is a plus.
- Strong problem-solving skills and attention to detail.
- Excellent communication and teamwork abilities.
Responsibilities
- Design, develop, and maintain scalable data pipelines and systems.
- Work closely with data scientists and analysts to ensure data quality and availability.
- Implement data integration and transformation processes using PySpark and Python.
- Optimize and maintain SQL databases and queries.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Monitor and troubleshoot data pipeline issues to ensure data integrity and performance.
Qualifications
Preferred Qualifications:
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- 3+ years of experience in data engineering.
- Proficiency in PySpark, Python, and SQL.
- Experience with cloud platforms such as Azure, AWS, or Google Cloud.
- Knowledge of data warehousing concepts and technologies.
- Familiarity with ETL tools and processes.
