Data Engineer

Responsibilities

• Build and maintain ETL/ELT pipelines in Microsoft Fabric Notebooks (Python/PySpark) ingesting data from multiple source systems via REST and SOAP APIs

• Implement Bronze layer landing (raw data, no transformation), Silver layer cleansing/typing/deduplication, and Gold layer aggression for business analytics

• Design and build custom API connectors with OAuth 2.0/Bearer token authentication, incremental sync, pagination handling, rate-limit/retry logic and error recovery

• Configure and manage Fabric Workspaces across Dev/Test/Production environments using Fabric Deployment Pipelines and Git integration for CI/CD

• Build and maintain Power BI semantic models (star/snowflake schemas) supporting operational reporting and analytics dashboards

• Implement row-level security (RLS) using Azure AD RBAC, warehouse-level DAX filters, and dynamic RLS platforms

• Set up monitoring, alerting, and telemetry using Azure Monitor, Log Analytics, and Application insights to track pipeline health and data freshness

• Manage API credentials and secrets via Azure Key Vault with automated rotation policies

• Embed within the client's development process, working directly with business stakeholders to gather requirements and co-develop solutions

• Contribute to data governance: data dictionary creation, lineage tracking, and documentation of transformation logic across all pipeline stages

Requirements

5+ years experience in data engineering or a similar role in a commercial environment

Hands-on experience with Microsoft Fabric (Lakehouses, Notebooks, Data Pipelines, OneLake, SQL Analytics Endpoints) or equivalent depth in Azure Synapse Analytics

Advanced SQL for data transformation, performance tuning, and Delta Lake table management

Proficient in Python/PySpark for data processing, API integration, and pipeline automation

Proven experience building custom API connectors (REST, SOAP/XML) with OAuth 2.0, pagination, rate limiting, and incremental sync patterns

Strong understanding of Medallion Architecture, data lakehouse concepts, and Delta Lake (merge, upsert, soft-delete handling, time travel)

Experience with dimensional modelling (star/snowflake schemas) and Power BI semantic model development

Working knowledge of Azure services: Key Vault, Azure Monitor, Log Analytics, Azure AD/Entra ID

Experience translating business requirements into dimensional models iteratively (not just building pre-designed schemas)

Comfort operating autonomously with minimal supervision in a client-embedded model

Experience implementing row-level security in Power BI (DAX-based RLS, dynamic security models)

Familiarity with CI/CD in a Fabric context: Deployment Pipelines, Git integration, environment promotion workflows

Experience with semi-structured data formats (JSON, XML) and handling schema evolution across pipeline layers

Preferred

Prior work in retail, wholesale, distribution, or fresh produce / supply chain data environments

Experience with ERP data extraction and replication tools in a Fabric or Azure context

Knowledge of dbt, Power Query, or similar transformation frameworks

Relevant Azure or Microsoft certifications (DP-600, DP-203, PL-300)

Openness to AI coding assistants (GitHub Copilot, Cursor, Claude) as part of your development workflow

What We Offer

• Competitive salary with mentorship and career growth opportunities

• Hands-on work with cloud-first technologies (Microsoft Fabric, Azure, Power BI)

• Long-term embedded role with a single client, providing deep domain expertise and continuity

• Fast-paced, innovative environment where AI-augmented development is the norm