Position: Mid-Senior Databricks Engineer
Location:
Remote
Experience:
- 3–6 Years Employment Type: Full-Time Required Skills & Qualifications 3–6 years of experience in Data Engineering.
- Strong knowledge of SQL, Python, and PySpark.
- Hands-on experience with Databricks and Apache Spark.
- Understanding of Spark optimization, partitioning, and shuffling.
- Experience with Delta Lake, Z-Ordering, and VACUUM.
- Experience building ETL pipelines using Delta Live Tables (DLT).
- Knowledge of Apache Airflow or Azure Data Factory.
- Understanding of data governance and data quality.
Key Responsibilities
- Develop and maintain scalable data pipelines using Databricks, Python, and PySpark.
- Build and automate ETL workflows using Delta Live Tables (DLT).
- Optimize Spark jobs through partitioning, shuffling, and performance tuning.
- Implement Delta Lake optimization techniques, including Z-Ordering and VACUUM.
- Manage data orchestration using Airflow or Azure Data Factory.
- Ensure data quality, governance, and pipeline performance.
- Collaborate with cross-functional teams to deliver efficient data engineering solutions.