M31 - Data Engineer

Job Overview We are seeking an experienced Data Engineer to design, develop and maintain scalable data pipelines, cloud-based data platforms and enterprise system integrations. The successful candidate will support data engineering, solution architecture, cloud integration and geospatial data processing initiatives. This role requires strong technical expertise, systems thinking and the ability to deliver secure, reliable and maintainable end-to-end solutions. You will collaborate with Data Managers, Business Analysts, Software Engineers, Cloud Engineers and other stakeholders to translate business requirements into practical technical solutions. Key Responsibilities Data Engineering & Pipeline Development • Design, develop, maintain and optimise data pipelines for structured, unstructured and geospatial datasets. • Develop data models, schemas, transformation logic and processing workflows for analytics and AI applications. • Implement automated data validation, quality monitoring and error-handling mechanisms. • Troubleshoot pipeline failures, resolve integration issues and improve system performance. • Maintain documentation covering data attributes, schemas, metadata and processing workflows. Solution Architecture & Systems Integration • Design and implement end-to-end solutions across cloud platforms, data systems, AI services and applications. • Integrate Azure Databricks, Azure cloud services, AI/ML platforms and geospatial applications. • Design data flows covering ingestion, transformation, storage, processing, APIs and downstream consumption. • Evaluate integration approaches such as REST APIs, database connections, batch processing and event-driven architectures. • Assess technical solutions based on scalability, security, performance, cost and maintainability. • Collaborate with software developers, cloud engineers and architects to define system interfaces and integration requirements. Cloud Data Platforms & AI Integration • Develop and maintain data engineering solutions using Azure Databricks and related Azure services. • Implement cloud-based data processing, orchestration, storage and integration solutions. • Support data integration with AI/ML services for model development, inference and AI-enabled applications. • Explore emerging AI technologies, including Retrieval-Augmented Generation (RAG), vector search and AI agents. • Collaborate with AI and platform teams to support evolving analytics and AI requirements. Geospatial Data Engineering • Develop geospatial data processing workflows, including spatial transformations, validation and integration. • Integrate backend data platforms with geospatial applications, mapping services and analytical tools. • Design efficient data access mechanisms using spatial databases, APIs and GIS services. • Optimise geospatial data processing to support interactive applications and analytics. • Evaluate geospatial technologies for storage, processing, visualisation and data sharing. Platform Operations, Security & DevSecOps • Support administration, maintenance and enhancement of data platforms across development, testing and production environments. • Perform platform upgrades, patching, configuration changes and environment management. • Implement monitoring, logging and troubleshooting processes to ensure platform reliability. • Conduct source code and configuration reviews to maintain secure coding practices. • Support authentication, authorisation, access management and secure data exchange. • Implement CI/CD pipelines, version control and deployment automation. • Ensure compliance with applicable security, governance and operational requirements. Technical Solutioning & Continuous Improvement • Analyse technical requirements and recommend suitable solutions. • Identify architectural gaps, integration challenges and performance bottlenecks. • Conduct technical assessments, proof-of-concepts and testing of emerging technologies. • Prepare architecture diagrams, data flow diagrams and integration specifications. • Communicate technical recommendations, risks and trade-offs to stakeholders. • Take ownership of technical issues and collaborate across teams to resolve them. Requirements • Degree in Computer Science, Computer Engineering, Information Systems, Software Engineering or a related discipline. • Experience designing, developing, deploying and maintaining data pipelines and cloud-based data platforms. • Strong understanding of solution architecture, systems integration and cloud infrastructure. • Experience integrating data platforms, cloud services and enterprise applications. • Familiarity with REST APIs, ETL/ELT processes, database connectivity and secure data exchange. • Understanding of data governance, security, identity and access management. • Strong troubleshooting, analytical and problem-solving skills. • Ability to translate business requirements into technical specifications and engineering solutions. • Good communication, documentation and stakeholder management skills. • Ability to manage multiple technical workstreams independently and collaboratively. #J-18808-Ljbffr