Capgemini - Pyspark/Azure Databricks Engineer

: We are looking for a skilled PySpark Developer with experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to join our team. The ideal candidate will play a crucial role in designing, developing, and implementing data solutions using PySpark for large-scale data processing and analytics. Responsibilities : - Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation. - Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources. - Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights. - Optimize PySpark jobs and data pipelines for performance, scalability, and reliability. - Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows. - Ensure data quality and integrity throughout all stages of data processing. - Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation. - Troubleshoot and debug issues related to data pipelines and processing. - Ensure compliance with regulatory requirements and industry standards in all data processing activities. - Implement best practices for data security, compliance, and privacy within Azure environment. - Document technical specifications, data flows, and solution architecture. Requirements : - Bachelors degree in computer science, Engineering, or a related field; Masters degree preferred. - Proven experience as a PySpark Developer or similar role with a strong understanding of Apache Spark internals. - Hands-on experience with Azure Databricks (ADB) and Azure Data Factory (ADF). - Proficiency in Python programming language and solid understanding of SQL. - Experience designing and optimizing data pipelines for ETL/ELT processes. - Familiarity with cloud platforms, preferably Microsoft Azure. - Excellent problem-solving skills and ability to think critically. - Strong communication skills with the ability to collaborate effectively in a team environment. - Experience in Financial, Risk, Compliance, or Banking domains is a plus. - Experience identifying and mitigating financial risks in data processes. - Ability to analyse data for potential risk factors and develop strategies to minimize financial risk. - Ensure all data processes comply with relevant regulatory requirements and industry standards. Preferred Qualifications : - Certification in Azure Data Engineering or related field. - Knowledge of other big data technologies such as Hadoop, Hive, or Kafka. - Familiarity with machine learning frameworks and techniques