L&T Technology - Data Engineer - ETL Tools

Key Responsibilities : - Design, develop, and maintain scalable data pipelines and ETL/ELT workflows. - Build and optimize data processing solutions using Databricks and cloud-based data platforms. - Develop data integration solutions using Azure Data Factory (ADF) and other ETL tools. - Work with structured and unstructured data from multiple data sources. - Develop and optimize data models, ensuring data quality, integrity, and consistency. - Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver scalable solutions. - Monitor, troubleshoot, and optimize data pipelines for performance and reliability. - Implement data governance, security, and best practices across the data platform. - Support data migration, transformation, and modernization initiatives. - Participate in code reviews, testing, deployment, and documentation activities. Required Skills : - Strong hands-on experience with Databricks. - Experience with Azure Data Factory (ADF) or equivalent ETL tools. - Proficiency in Azure and/or AWS cloud platforms. - Good understanding of Hadoop ecosystem and distributed data processing. - Strong experience in designing and developing ETL/ELT pipelines. - Proficiency in Python, SQL, or Scala for data engineering. - Experience with relational and NoSQL databases. - Knowledge of data warehousing concepts and dimensional data modeling. - Familiarity with version control tools such as Git. - Strong analytical, debugging, and performance optimization skills. Preferred Skills: - Experience with Apache Spark and PySpark. - Knowledge of Delta Lake, Apache Kafka, or streaming data pipelines. - Exposure to CI/CD pipelines using Azure DevOps, Jenkins, or similar tools. - Familiarity with containerization technologies such as Docker and Kubernetes. - Experience with data governance and cloud security best practices. Educational Qualification: - B.E./B.Tech in Computer Science, Information Technology, or a related discipline. - MCA or M.Sc. in Computer Science, Information Technology, or a related field is also acceptable.