Data Engineer - Python/Azure Databricks

Role & Responsibilities : Key Responsibilities : - Architect and implement enterprise-grade Lakehouse solutions using Databricks. - Design and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL). - Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworks. - Implement and optimize Databricks components: Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and Workflows. - Design large-scale data warehousing solutions with 3NF and dimensional modeling. - Establish data governance, security, and data quality frameworks, including Unity Catalog. - Lead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI). - Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow). - Drive CI/CD, DevOps practices, and performance optimization of Spark workloads. - Provide technical leadership, mentor teams, and ensure successful delivery. - Collaborate with stakeholders to translate business requirements into scalable solutions. Ideal Candidate : - Strong Databricks Architect Profile with end-to-end Lakehouse ownership. - Mandatory (Experience 1) : Must have 10+ years of software engineering experience with at least 5+ years in Data Engineering with hands on exposure to Databricks and strong ownership of end-to-end data pipeline development. - Mandatory (Experience 2) : Must have at least 5+ years of expertise across the Databricks ecosystem Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, Workflows, Unity Catalog. - Mandatory (Tech skill 1) : Must have worked at architecture level, owning end-to-end design through deployment. - Mandatory (Tech skill 2) : Must have strong experience with Python and SQL for data processing and Apache Spark for performance tuning & scalability. - Mandatory (Tech skill 3) : Must have experience in large-scale data warehousing & advanced data modeling (3NF and dimensional) across batch and real-time systems. - Mandatory (AI Exposure) : Must have at least a basic working understanding of how AI services or tools work. - Mandatory (Communication) : Must have strong stakeholder management & requirement-gathering experience with US or UK clients. - Mandatory (Company) : Must come from a B2B IT services or IT consulting background. - Mandatory (Note 1) : CTC is inclusive of 5% variable. - Preferred (Tech skill 1) : Azure Databricks or Azure data services experience (project runs on Azure DevOps). - Preferred (Tech skill 2) : MLflow or MLOps practices and AI use cases (RAG, AI/BI). - Preferred (Tech skill 3) : CI/CD, Databricks Asset Bundles (DABs) or equivalent packaging, Terraform or IaC, reusable deployment templates. - Preferred (Integrations) : ServiceNow or enterprise system integrations. - Preferred (Certifications) : Databricks (Data Engineer Associate or Professional, ML or GenAI tracks), Azure or AWS cloud certifications. Perks, Benefits and Work Culture : - The company provides free AWS and Azure certification training. - Group medical insurance of 5 lakhs is included in the benefits package, along with meal allowances.