InfoBeans Technologies - Data Engineer - Azure Databricks

About The Role We are looking for an experienced Data Engineer with strong hands-on expertise in Python, PySpark, SQL, and cloud-based data engineering. The role involves designing and developing scalable data processing solutions, building reusable frameworks, optimizing data transformation pipelines, and delivering reliable data solutions using Azure Databricks and Azure Data Factory (ADF). Key Responsibilities • Design, develop, and maintain scalable data processing solutions using Python and PySpark. • Write complex SQL queries for data extraction, transformation, validation, reconciliation, and analysis. • Build and maintain robust data pipelines and workflows using Azure Databricks and Azure Data Factory (ADF). • Develop reusable, modular, and scalable code frameworks to support efficient data engineering. • Design and implement data transformations and processing logic for large datasets. • Develop solutions from scratch, following clean coding practices, reusable design patterns, and comprehensive documentation standards. • Optimize existing code, Spark jobs, SQL queries, and data pipelines to improve performance, scalability, and resource utilization. • Troubleshoot data pipeline failures, processing bottlenecks, and data quality issues. • Collaborate with data engineers, analysts, architects, and business stakeholders. • Ensure data accuracy, consistency, integrity, and reliability throughout the data processing lifecycle. • Participate in technical design discussions, code reviews, and continuous improvement initiatives. Required Skills & Experience • 10 - 13 years of experience in data engineering, data development, or related software engineering roles. • Strong hands-on proficiency in Python, including core programming concepts, data structures, modular design, and reusable code development. • Strong experience with PySpark for distributed data processing and large-scale data transformations. • Advanced SQL skills, including complex queries, joins, aggregations, analytical functions, and query optimization. • Hands-on experience with Azure Databricks for data engineering and transformation workloads. • Experience building and orchestrating data pipelines using Azure Data Factory (ADF). • Strong understanding of data engineering principles, ETL/ELT processes, data transformation, and pipeline orchestration. • Experience designing scalable and reusable data processing frameworks. • Experience working with Git, code reviews, testing, and deployment practices. • Excellent analytical, communication, and cross-functional collaboration skills. (ref:hirist.tech)