Senior Data Platform Engineer | Data Integration & Modernisation

Job Responsibilities Lead the architecture, design and implementation of ETL processes, workflows and data pipelines for the ingestion, transformation and integration of data from multiple source agencies.

Build and maintain efficient and reliable large-scale batch and real-time data pipelines, including dependency management, failure handling, recovery and operational monitoring.

Develop and maintain modular, production-quality code for statistical production processes, including the modernisation of legacy processes while preserving existing business logic and reconciling outputs against current production results.

Review complex business logic and provide technical direction for data migration and modernisation initiatives. Lead database and data model design, including schema design, storage patterns and performance optimisation for large-scale statistical datasets.

Design and drive the adoption of reusable data engineering components, including common libraries, configuration frameworks, logging standards and utilities to streamline production workflows and promote consistent engineering practices.

Design and implement CI/CD pipelines for data engineering workflows to automate testing and deployment, while establishing workflows across Development, UAT and Production environments.

Lead the design and configuration of complex pipeline workflows and schedules using enterprise tools, including dependency management, failure handling, recovery and operational monitoring.

Establish version control structures using GitLab, including branching strategies, merge requests and code review frameworks, while maintaining technical architecture and implementation documentation.

Provide technical reviews for solution designs, merge requests and implementation approaches, and promote appropriate engineering standards across the development team. Troubleshoot production incidents, monitor pipeline health and continuously optimise system performance for large-scale datasets. Lea