Senior Data Engineer Consultant – Data Quality and Modern Cloud Platform

Function: Data & Digital Consulting
Location: South Africa, hybrid / client-site as required
Reports to: Data Engineering or Consulting Practice Lead

About Keyrus

Keyrus is an internationally recognised specialist in Data and Digital, and a trusted partner to organisations across industries. We deliver practical business solutions using modern, scalable technologies, helping clients improve performance through data-enabled transformation.

We're passionate about innovation, teamwork and collective success, driven by exceptional people who combine technical depth with sound consulting judgement.

The Role

The Senior Data Engineer Consultant designs, builds and operationalises secure, scalable, governed data platforms for Keyrus clients. You'll turn business and data requirements into reliable ingestion, transformation, quality, storage and serving solutions that power reporting, analytics, migration and responsible AI use cases.

You'll work closely with data architects, business analysts, governance teams and client stakeholders, providing technical leadership, shaping solution architecture, supporting pre-sales and mentoring other engineers.

Key outcomes

• Secure, scalable, reliable and cost-conscious cloud data platforms and pipelines.
• Trusted, curated, reporting-ready data products with clear ownership and measurable quality.
• Repeatable profiling, cleansing, remediation, reconciliation and monitoring capabilities.
• Traceable metadata, lineage and transformation logic that support governance and sign-off.
• AI-ready data foundations and responsible use of AI-assisted engineering tools.
• Runbooks, knowledge transfer and a smooth transition into business-as-usual support.

What you'll be doing

Discovery and design

Assess client systems and data flows, document current-state architecture, and translate requirements into technical designs, mappings and delivery backlogs. Contribute to target-state lake, lakehouse and warehouse architecture, flag risks early, and explain trade-offs clearly to any audience.

Platform engineering
Build reusable ingestion, transformation and orchestration frameworks across databases, APIs, files, SaaS and streaming sources. Implement batch, incremental and CDC patterns with proper reconciliation, apply strong engineering practices (peer review, testing, CI/CD), and optimise SQL and Spark for performance and cost.

Data quality and remediation

Profile complex datasets, quantify duplicates and anomalies, and build validation, matching and deduplication rules. Develop version-controlled, auditable cleansing pipelines with exception handling and reconciliation, working with data owners to sign off fixes.

Governance, security and privacy
Implement metadata, catalogue and end-to-end lineage, maintaining clear mappings and change records. Apply least-privilege access, encryption and masking, monitor data access and pipeline activity, and handle sensitive data in line with POPIA and client controls.

Operations and migration
Build in observability, retry and recovery, define service levels and escalation paths, and produce clear runbooks and deployment guides. Support migration through profiling, reconciliation and sign-off evidence, and run structured knowledge transfer into client teams.

Responsible AI and learning
Design data foundations that support analytics and AI use cases responsibly, using only approved tools. Ensure AI-assisted work is tested and reviewed, never expose client data to unapproved AI services, and stay current on AI risks, sharing what you learn through mentoring and reusable standards.

Consulting and leadership
Lead workstreams from discovery through to handover, plan and estimate realistically, and escalate risk early. Define evidence-based acceptance criteria, mentor engineers, conduct reviews, and contribute to proposals and pre-sales. Translate technical work into clear business value and build trusted client relationships.

What you'll bring

Education
A relevant degree or diploma (computer science, information systems, engineering, data science or similar), or equivalent hands-on experience. Microsoft, Databricks, cloud architecture or data-engineering certifications are a plus.

Essential experience

• Seven or more years in data engineering or data platforms, including three or more years delivering production-grade cloud solutions.
• Advanced SQL, plus strong Python or PySpark skills.
• Hands-on with Azure Data Factory or Fabric Data Factory, Azure Data Lake Storage and Azure Databricks, or close equivalents.
• Solid grasp of lake, lakehouse and warehouse architecture and modelling.
• Delivery experience across ingestion, transformation, orchestration and source-system sync.
• Experience profiling, cleansing and reconciling large or messy datasets.
• Working knowledge of entity matching, exception handling and quality controls.
• Experience with metadata, catalogue and lineage tools.
• Cloud security, IAM, secrets management, encryption and audit logging know-how.
• Comfortable with Git, automated testing, CI/CD and environment management.
• A track record of technical leadership, client engagement and clean handovers.

Desirable experience

• Microsoft Fabric, Delta Lake and medallion architecture.
• Microsoft Purview or similar governance and lineage tools.
• Data-quality frameworks such as Great Expectations, Soda or dbt tests.
• Infrastructure as code (Bicep, Terraform or equivalent).
• Power BI semantic models and reporting-ready data products.
• SQL Server, T-SQL, SSIS, SSAS, SSRS and PowerShell in legacy or hybrid settings.
• API integration, event-driven or streaming architectures.
• Large-scale migration, reconciliation and cutover support.
• Experience in regulated data domains (customer, billing, banking and similar).
• Applied knowledge of generative AI, RAG, embeddings and vector stores.

How you work

• Analytical: gets to root causes and picks proportionate, evidence-based solutions.
• Disciplined: builds maintainable, tested, secure solutions, not one-off scripts.
• Clear communicator: adapts easily between executives, business owners and engineers.
• Ownership-driven: plans realistically and sees work through to acceptance.
• Collaborative: works well across architecture, governance and client teams.
• Commercially aware: understands cost, licensing and the impact of technical choices.
• Always learning: stays current and turns new knowledge into repeatable practice.
• Trustworthy: protects client information and is upfront about risks and limitations.

Role boundaries

This is a senior data engineering role within a multidisciplinary delivery team. You'll collaborate closely with data architecture, data quality, governance, business analysis, analytics and project leadership specialists. You may lead a data-engineering workstream, but this role isn't intended to cover every specialist discipline on a complex enterprise data programme.

Success measures

• Solutions meet agreed functional, quality, security, performance and acceptance requirements.
• Pipelines and data products run reliably within agreed service levels and cost.
• Data rules, changes, lineage and remediation outcomes are traceable and auditable.
• Client teams can support and extend delivered solutions using the documentation and knowledge transferred.
• Technical risks and delivery constraints are flagged early and managed transparently.
• Reusable assets and lessons learned improve future Keyrus delivery quality.

Employment equity

Keyrus is committed to employment equity and to creating an inclusive workplace. Applications from suitably qualified candidates are welcomed, with due regard to achieving equity in race, gender and disability representation.

How to apply

Please submit a CV highlighting relevant cloud data-platform implementations, data-quality or migration programmes, technical leadership responsibilities, and the scale and business outcomes of solutions you've delivered.