Happiest Minds Technologies - Data Engineer - ETL/Python

Role Overview : We are looking for a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data platforms. The role will be responsible for building reliable data ingestion and transformation processes, optimizing data workflows, and enabling high-quality data availability for analytics and business applications. Key Responsibilities : 1. Data Pipeline Development : - Design, develop, and maintain scalable batch and real-time data pipelines. - Build robust ETL/ELT workflows for structured and unstructured data. - Integrate data from multiple internal and external sources. - Develop reusable data processing frameworks and transformation logic. - Ensure reliable and timely availability of data for downstream consumers. 2. Data Platform & Architecture : - Design scalable data models and data processing solutions. - Work with data lakes, data warehouses, and distributed data platforms. - Optimize storage, processing, and data retrieval mechanisms. - Contribute to data architecture and platform modernization initiatives. - Ensure data platforms are scalable, secure, and highly available. 3. Data Quality & Governance : - Implement data validation, quality checks, monitoring, and reconciliation processes. - Identify and resolve data inconsistencies and pipeline failures. - Maintain data lineage and appropriate technical documentation. - Follow organizational standards for data security, governance, and access control. 4. Performance & Reliability : - Monitor pipeline performance and troubleshoot production issues. - Optimize queries, transformations, and data processing jobs. - Implement appropriate logging, alerting, and failure-recovery mechanisms. - Improve pipeline efficiency and reduce processing time and infrastructure costs. Required Skills & Qualifications : - 5 - 10 years of experience in Data Engineering or related roles. - Strong programming experience in Python, SQL, or similar languages. - Strong understanding of ETL/ELT concepts and data pipeline development. - Experience with data warehouses, data lakes, and distributed processing technologies. - Experience with Apache Spark/PySpark is preferred. - Knowledge of cloud platforms such as AWS, Azure, or GCP. - Experience with orchestration tools such as Airflow or similar platforms. - Strong understanding of data modelling, database concepts, and performance optimization. - Good understanding of Git, CI/CD, testing, and production support practices.