Lead Data Engineer

• Design and Develop Data Pipelines: Hands on with development, and optimisation of scalable and reusable - data pipelines in Azure Microsoft Fabric Synapse Data Engineering , leveraging both batch and real-time processing techniques . Ensure smooth integration with Azure Data Factory for orchestration and workflow management. • Cloud Data Architecture: Collaborate with the Data Architecture team to design and implement robust data architectures in the Azure environment , ensuring they align with business needs while optimising performance, scalability, and cost-efficiency. • Pipeline Optimisation : Continuously monitor and optimise the performance, cost, and reliability of data pipelines, ensuring efficient processing, storage, and management of large datasets. • Cross-functional Collaboration: Work closely with data engineering teams , analysts , and business stakeholders to understand data requirements, developing solutions that enable self-service analytics and support the decision-making process. • Documentation Knowledge Sharing: Contribute to internal documentation, fostering a culture of knowledge-sharing. Provide mentorship and guidance to junior engineers, helping to elevate team skills and improve overall team performance. • Microsoft Fabric Experience: Apply your knowledge of Azure Tech Stack on Data Engineering ( or your willingness to learn ) Fabric-based development to manage end-to-end data orchestration, governance, and security across cloud and on-premises systems, ensuring seamless data movement and integration across hybrid environments. • Data Modelling Expertise: Leverage your deep expertise in Azure to design and implement data models , create processing pipelines, and integrate with other Azure services like Data Lake and Synapse to support data storage and analytics needs. Required Skills and Qualifications: • Experience with Azure Ecosystem (Preferably Synapse): 5+ years of hands-on experience with Azure Ecosystem , including Synapse , Spark , OneLake , and other Fabric tools. Expertise in optimising Fabric notebooks and efficiently managing large-scale data workloads. • Proficiency in Azure Data Factory: Strong experience with designing and orchestrating complex data pipelines using Azure Data Factory , with an emphasis on seamless data flow integration across various Azure services. • Familiarity with Microsoft Fabric: A working knowledge or eagerness to learn Azure Data Fabric , focusing on cross-platform data orchestration, governance, and security. • Advanced Data Engineering Skills: Extensive experience in data engineering, including the design and implementation of ETL processes and working with large datasets. Proven expertise in data quality , monitoring, and testing practices. • Cloud Architecture Design Expertise: Experience designing and implementing data architectures in the Azure ecosystem , including tools such as Data Lake , Synapse , and Azure Storage . • SQL and Data Modelling Expertise: Strong skills in SQL and data modelling , with the ability to design optimised data structures, tables, and views. Knowledge of both transactional and analytical data modelling. • Collaboration and Communication Skills: Strong ability to work cross-functionally with teams from various domains. Ability to communicate complex technical concepts to both technical and non-technical stakeholders. • Cost Optimisation: Proven experience optimising data engineering processes and Azure resources for both performance and cost, particularly in large-scale cloud environments. Preferred Skills: • Data Lakehouse Experience: Familiarity with Data Lakehouse architectures, particularly with tools like Delta Lake , OneLake , etc. • Azure Ecosystem Familiarity: Knowledge of Azure s full ecosystem for end-to-end data integration and ETL processes. • Proficiency in PySpark and Python: Expertise in PySpark for data processing tasks, with a solid foundation in Python . • Fabric Integration: Familiarity with Fabric and how it integrates with other services within the Azure ecosystem . • Databricks Experience: Experience with Databricks is a plus. Skills: Pyspark, data engineering , Azure, Databricks, Python Experience: 5.00-10.00 Years