Data Engineer

Job Summary We are looking for an experienced Data Engineer to design, build, and maintain a scalable, metadata-driven data ingestion and transformation framework on Google Cloud Platform (GCP) .

The role will be responsible for developing configuration-driven ingestion pipelines, onboarding multiple source systems, and implementing Raw, Bronze, Silver, and Gold data layers .

The Data Engineer will use technologies including BigQuery, Cloud Composer/Apache Airflow, Python, Dataform, Datastream, Pub/Sub, Dataflow, Cloud Run, and Cloud Storage to deliver reliable and reusable data solutions.

The ideal candidate will have strong experience in metadata-driven pipeline development, CDC, advanced SQL, Python, data quality, reconciliation, automated testing, and CI/CD .

The engineer will work closely with Data Governance and other platform stakeholders to ensure that data pipelines are scalable, governed, auditable, testable, and production-ready.

Responsibilities

Design and build a metadata-driven and configuration-driven ingestion framework using reusable pipeline templates. Design, develop, and maintain the metadata configuration model covering source systems, source objects, load configurations, execution parameters, run logs, audit information, and reprocessing.

Build dynamic Cloud Composer / Apache Airflow DAGs that generate and execute ingestion workflows based on metadata configurations. Develop reusable ingestion patterns and pipeline generators instead of building separate pipelines for individual source tables.

Onboard multiple source systems, including relational databases, SAP, REST APIs, files, streaming sources, and other enterprise platforms . Implement data ingestion into Raw and Bronze layers following defined lakehouse architecture and engineering standards.

Implement source-to-target reconciliation to ensure completeness and accuracy of ingested data. Implement robust operational controls, including error handling, retries, quarantine processing, alerti