Senior Data Engineer (Databricks)

About the Role We are looking for an experienced Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines that enable data-driven decision-making.

You will work closely with our client's stakeholders to develop reliable, secure, and high-performance data solutions while driving best practices in Data Engineering, CI/CD, and Site Reliability Engineering (SRE).

This role is ideal for engineers who are passionate about building modern data platforms, ensuring data quality, and delivering resilient cloud-based data solutions.

Key Responsibilities

Data Pipeline Development & Architecture Translate business data requirements into technical solutions. Design, develop, and maintain scalable batch and real-time data pipelines. Build data ingestion pipelines to collect, cleanse, transform, and integrate data from multiple source systems.

Design reusable, scalable, and optimized data models to support analytics and reporting. Develop secure mechanisms for users and applications to access enterprise data. Research, evaluate, and implement new technologies to improve the organization's data infrastructure.

Data Quality, Testing & Validation Design and implement data quality frameworks to ensure data accuracy, completeness, and consistency. Develop and execute functional, regression, and exploratory testing for data pipelines. Build automated testing capabilities to improve deployment quality and reduce production defects.

Perform data reconciliation and end-to-end validation across multiple systems. Support User Acceptance Testing (UAT) and collaborate with business stakeholders to validate solutions.

Operations, Governance & Continuous Improvement Monitor and maintain databases and ETL platforms, including performance tuning, capacity planning, troubleshooting, and issue resolution. Collaborate with infrastructure and security teams to ensure scalable and secure data platforms.

Work with data stewards to establish and enforce data governance sta