Senior Data Engineer - #1732

Key Responsibilities

• Data Pipeline Architecture and Delivery: Lead the design and implementation of maintainable batch and streaming pipelines that ingest, validate, transform and publish structured, semi-structured and unstructured data across AWS and Databricks.

• Lakehouse and Data Product Engineering: Design Bronze, Silver and Gold data layers using Delta Lake or equivalent patterns; develop reusable domain data models, curated datasets and data products with explicit ownership, definitions and service expectations.

• Source Integration and Incremental Processing: Implement reliable ingestion from databases, files, APIs and event streams using full-load, incremental, change-data-capture and replayable processing patterns appropriate to each source.

• Data Quality and Contracts: Establish schema enforcement, data contracts, automated validation, reconciliation and anomaly checks covering completeness, validity, consistency, uniqueness, timeliness and referential integrity.

• Orchestration and Deployment Automation: Build parameterised workflows using Databricks Workflows or Lakeflow and relevant AWS services; integrate automated testing, nfrastructure as code and CI/CD controls across development, test and production environments.

• Metadata, Lineage and Governance: Implement technical metadata, business definitions, end-to-end lineage, classification, ownership and access controls using Unity Catalog, AWS Glue Data Catalog, Lake Formation or approved platform capabilities.

• Pipeline Monitoring and Reliability: Define dashboards, alerts, logging, operational metrics and runbooks for pipeline health, freshness, data-quality failures, latency, throughput and downstream impact; lead troubleshooting and root-cause analysis.

• Performance and Cost Optimisation: Optimise Spark and SQL workloads, file sizes, partitioning, clustering, caching, compaction, cluster configuration and workload scheduling to improve runtime, reliability and AWS/Databricks cost efficiency.