Senior Data Engineer (Databricks | Data & AI)
Location:
Indianapolis, IN Position Overview We are seeking a senior hands-on Data Engineer to design, develop, and scale enterprise data platforms supporting data and AI initiatives for our client. The ideal candidate has deep experience building on AWS and Databricks, and can work directly with business and technical stakeholders, understand emerging requirements, and identify opportunities for additional data and AI solutions.
This role supports one of our client's business units, with potential for the engagement to expand.
Key Responsibilities
- Design, build, and scale data pipelines on Databricks and AWS.
- Develop ETL/ELT solutions using Spark, PySpark, and SQL.
- Implement Delta Lake, Lakehouse, Unity Catalog, and Databricks Workflows.
- Work with AWS data services such as S3, Glue, Lambda, Redshift, and IAM.
- Build batch and streaming ingestion with data quality and governance controls.
- Apply CI/CD practices and optimize pipeline performance, cost, and reliability.
- Engage business and technical stakeholders directly and translate requirements into solutions.
- Contribute to data architecture and solution design discussions.
- Identify opportunities where data and AI can address additional business needs.
- Contribute to the growth of the overall engagement.
- Requirements Required Qualifications Strong hands-on data engineering background.
- Python, PySpark/Spark, and SQL.
- Databricks, including Delta Lake and Lakehouse.
- Strong hands-on experience with AWS data services.
- ETL/ELT pipeline development and data modelling.
- Batch and streaming ingestion, data quality and governance, CI/CD.
- Ability to engage stakeholders independently, not only work from technical tickets.
- Excellent communication and stakeholder management skills.
- Must be local to Indianapolis, IN and able to work onsite.
- Preferred Experience Unity Catalog, MLflow, Medallion Architecture Databricks certification Azure NVIDIA ecosystem exposure GenAI and ML pipelines Regulated healthcare or pharma experience (pharma preferred, not mandatory) Experience identifying additional technology or business opportunities
