Data Engineer - Azure Data Factory

Role: Data Engineer Experience: 5-10 yrs Job location: Trivandrum We are seeking a skilled Azure Data Engineer with 510 years of experience in designing, developing, and maintaining modern data engineering solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure Data Factory (ADF), Azure Databricks (ADB), and Azure DevOps, with experience building scalable data pipelines, data transformation workflows, and cloud-based analytics platforms. Key Responsibilities: - Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF). - Build and optimize data processing workflows using Azure Databricks (PySpark/Scala/SQL). - Develop ETL/ELT solutions to ingest, transform, and load data from multiple source systems. - Implement data integration solutions for structured, semi-structured, and unstructured data. - Collaborate with business analysts, data architects, and stakeholders to understand data requirements. - Optimize data workflows for performance, scalability, and cost efficiency. - Implement CI/CD pipelines using Azure DevOps for data engineering solutions. - Monitor, troubleshoot, and resolve data pipeline failures and performance issues. - Ensure data quality, security, governance, and compliance with organizational standards. - Participate in code reviews and promote engineering best practices. - Create and maintain technical documentation for data pipelines and workflows. Required Skills: - 510 years of experience in data engineering or data integration. - Strong hands-on experience with Azure Data Factory (ADF). - Expertise in Azure Databricks (ADB) using PySpark, Spark SQL, or Scala. - Experience with Azure DevOps, including Git repositories, Boards, Pipelines, and CI/CD implementation. - Strong SQL skills with experience in query optimization and database design. - Experience working with Azure Data Lake Storage (ADLS) and cloud-based data storage solutions. - Good understanding of ETL/ELT concepts and data warehousing principles. - Experience integrating data from APIs, databases, files, and cloud services. - Strong analytical, troubleshooting, and problem-solving skills. - Excellent communication and collaboration abilities.