Data Engineer - #1694

Role Overview The Data Engineer works closely with statisticians, analytical teams, and business stakeholders to design, build, and maintain the data infrastructure needed to support rigorous statistical production processes.

The role centres on building reliable and maintainable data pipelines, validation and frameworks that support statistical production processes, while providing technical leadership across the engineering team.

You will work closely with the user team on the business needs, translate them into robust technical solutions and lead in the modernisation of legacy processes. This includes ensuring that existing business logic is preserved and that outputs are accurately reconciled against current production results.

In addition, you will provide technical guidance on solution design, engineering standards, and implementation approaches to ensure the team delivers production-quality solutions.

Key Responsibilities

Leads the architecture, design, and implementation of ETL processes, workflows, and data pipelines for the ingestion, transformation, and integration of data from multiple source agencies.

Build and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks, including dependency management, failure handling, recovery, and operational monitoring.

Develop and maintain modular, production-quality code for statistical production processes, including uplifting of legacy processes while preserving business logic and reconciling outputs against existing results. Review complex business logic and provide technical direction for migration and modernisation activities.

Lead database and data model design, including schema design, storage patterns and performance optimisation for large-scale statistical datasets.

Design and drive the adoption of reusable data engineering components such as common libraries, configuration frameworks, logging standards, and utilities to streamline production workflows a