DysrupIT - Senior Databricks Engineer - Python/SQL

We are seeking a highly skilled Databricks Engineer with deep experience in Unity Catalog and enterprise-scale data platforms to support a major Oil & Gas Hydrocarbon Reporting initiative. The engineer will augment an existing delivery team and contribute across data engineering, governance, optimization, and analytics initiatives on the Databricks Lakehouse Platform. The ideal candidate has hands-on experience implementing secure, governed data architectures using Unity Catalog and has worked on large-scale regulatory, production, or hydrocarbon reporting programs within the energy sector. Role & responsibilities : - Design, develop, and maintain scalable data pipelines on the Databricks Lakehouse Platform. - Implement and administer Unity Catalog for enterprise data governance, access control, lineage, and data sharing. - Support hydrocarbon reporting workflows, including ingestion, transformation, validation, and reconciliation of production and operational data. - Develop ETL/ELT pipelines using Spark, Delta Lake, and Delta Live Tables (Lakeflow experience is a plus). - Optimize Databricks workloads for performance, reliability, and cost efficiency. - Collaborate with business stakeholders to translate reporting requirements into scalable technical solutions. - Implement data quality checks, monitoring, and automated validation frameworks. - Work closely with solution architects and data engineers on platform enhancements and production support. - Document technical designs, governance standards, and operational procedures. - Participate in code reviews, CI/CD implementation, and DevOps best practices. Required Qualifications : - Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related discipline. - 5+ years of data engineering experience with 3+ years of hands-on Databricks experience. - Strong experience implementing and managing Unity Catalog in enterprise environments. - Experience supporting Oil & Gas, Energy, or Hydrocarbon Reporting projects. - Expertise with : Databricks Lakehouse, Apache Spark (PySpark), Delta Lake, SQL, Python. - Experience building enterprise ETL/ELT pipelines. - Experience with Azure Databricks (AWS Databricks also considered). - Strong understanding of data governance, security, metadata management, and lineage. Preferred Qualifications : - Experience with Delta Live Tables (DLT) or Lakeflow Pipelines. - Knowledge of Unity Catalog features including ABAC, row/column-level security, external locations, catalogs, schemas, and lineage. - Experience integrating with Azure Data Factory, Event Hubs, Kafka, or similar ingestion frameworks. - Familiarity with MLflow, Databricks Workflows, and Databricks Asset Bundles. - Experience working with hydrocarbon production, drilling, refinery, pipeline, or regulatory reporting datasets. - Understanding of DevOps, Git, CI/CD, and Infrastructure as Code (Terraform preferred).