Senior Data Engineer (AWS & Databricks) | Contract

Key Responsibilities • Design, develop and maintain scalable batch and streaming data pipelines across AWS and Databricks. • Build and manage lakehouse architectures using Apache Spark, PySpark and Delta Lake, including Bronze, Silver and Gold data layers. • Integrate data from databases, APIs, files and event streams using incremental ingestion and change data capture. • Implement data quality checks, schema validation, reconciliation, metadata management and data lineage. • Develop automated workflows and CI/CD pipelines using Databricks Workflows, AWS services, Git and infrastructure-as-code tools. • Monitor pipeline performance, troubleshoot issues and implement logging, alerts and recovery procedures. • Optimise Spark and SQL workloads for performance, reliability and cloud cost efficiency. • Implement data security, access controls, encryption and governance in line with agency requirements. • Establish engineering standards, review code and architecture, and mentor junior engineers. • Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions. • Maintain technical documentation, operating procedures and knowledge-transfer materials. Requirements & Qualifications • Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related field. • 6-10 years of relevant data or software engineering experience, including at least 2 years working with cloud-based data platforms. • Strong hands-on experience in Python, PySpark, SQL, Apache Spark and Delta Lake. • Proven experience building and operating production data pipelines using Databricks and AWS. • Good knowledge of AWS services such as S3, Glue, Lake Formation, IAM, Lambda, Redshift and CloudWatch. • Experience with lakehouse architecture, data modelling, incremental processing, change data capture and data quality management. • Familiarity with workflow orchestration, CI/CD, Git and infrastructure-as-code tools such as Terraform. • Experience with pipeline monitoring, troubleshooting, performance optimisation and cloud security. • Ability to lead technical design, conduct code reviews, establish engineering standards and mentor engineers. • Strong problem-solving, communication and stakeholder management skills. Preferred Skills • Experience with Unity Catalog, Databricks Workflows, Lakeflow, Delta Live Tables or Structured Streaming. • Familiarity with Airflow, dbt, Kafka, AWS DMS, MSK or Kinesis. • Experience building data products, data contracts, feature engineering or ML data pipelines. • Knowledge of Singapore public-sector data governance, security and compliance requirements. • Relevant Databricks or AWS certifications. #J-18808-Ljbffr