Azure Data Engineer - PySpark/Databricks

About the Role : We are looking for an experienced Azure Data Engineer with 6 - 15 years of hands-on experience in designing, developing, and implementing scalable data engineering solutions on the Microsoft Azure platform. The ideal candidate should have strong expertise in Azure Databricks, Azure Data Factory, and PySpark, along with good exposure to Azure Synapse, Logic Apps, and ADLS. The candidate will be responsible for building robust data pipelines, transforming and processing large datasets, optimizing data workflows, and supporting enterprise-scale data platforms. Key Responsibilities : - Design, develop, and maintain scalable and reliable data pipelines using Azure Data Factory and Azure Databricks. - Develop complex data transformations and processing workflows using PySpark. - Work extensively with Azure Databricks for data ingestion, transformation, processing, and analytics. - Build and orchestrate ETL/ELT pipelines using Azure Data Factory (ADF). - Develop reusable and optimized PySpark code for processing large volumes of structured and semi-structured data. - Work with Azure Data Lake Storage (ADLS) for storing and managing enterprise data. - Integrate data from multiple sources including databases, APIs, files, and cloud-based applications. - Design data workflows using ADF pipelines, datasets, linked services, triggers, parameters, and integration runtimes. - Work with Azure Synapse Analytics for data warehousing, analytics, and large-scale data processing. - Develop and integrate workflows using Azure Logic Apps where required. - Implement data quality, validation, error handling, and monitoring mechanisms across data pipelines. - Optimize Databricks jobs, Spark workloads, SQL queries, and data pipelines for performance and cost efficiency. - Troubleshoot failures and performance issues across Azure data engineering solutions. - Implement appropriate security and access controls for data and Azure resources. - Collaborate with data architects, analysts, developers, and business stakeholders to understand data requirements. - Participate in code reviews and follow development, testing, deployment, and documentation standards. - Support production deployments and provide troubleshooting for critical data pipeline issues. - Contribute to the continuous improvement and modernization of existing data platforms. Mandatory Technical Skills : - Azure Databricks - Strong hands-on experience - Azure Data Factory (ADF) - Strong hands-on experience - PySpark - Strong programming and data processing experience - Strong understanding of ETL/ELT concepts and data pipeline development - Experience working with Azure Data Lake Storage (ADLS) - Good knowledge of Azure Synapse Analytics - Exposure to Azure Logic Apps - Strong understanding of data transformation, ingestion, integration, and processing concepts. - Experience working with large datasets and distributed data processing. - Experience with Azure DevOps and CI/CD pipelines - Knowledge of SQL and relational databases - Experience with Delta Lake / Delta tables - Knowledge of Apache Spark - Experience with data warehousing concepts and dimensional modelling. - Understanding of cloud security, access management, and Azure governance. - Experience working in Agile/Scrum environments.