Senior Data Engineer - AWS/Python

Roles & Responsibilities : This role offers an exciting opportunity to work on diverse projects, collaborating with cross functional teams to design, build, and optimize data pipelines and infrastructure. Responsibilities : - Design, develop, and maintain scalable data pipelines and ETL processes leveraging AWS services such as S3, Glue, EMR, Lambda, and Redshift. - Collaborate with data scientists and analysts to understand data requirements and implement solutions that support analytics and machine learning initiatives. - Optimize data storage and retrieval mechanisms to ensure performance, reliability, and cost-effectiveness. - Implement data governance and security best practices to ensure compliance and data integrity. - Troubleshoot and debug data pipeline issues, providing timely resolution and proactive monitoring. - Stay abreast of emerging technologies and industry trends, recommending innovative solutions to enhance data engineering capabilities. Ideal Candidate : - 1. Strong Data Engineer (AWS) Profile : - 2. Mandatory (Experience 1): Must have minimum 6+ years of experience in Data Engineering, with strong experience designing, developing, and maintaining scalable data pipelines and ETL processes. - 3. Mandatory (Experience 2): Must have 3+ years of hands-on experience with AWS data engineering services, particularly S3, AWS Glue, EMR, Lambda, and Redshift. - 4. Mandatory (Experience 3): Must have strong programming experience in Python, Java, or Scala, with hands-on development of data processing and pipeline solutions. - 5. Mandatory (Experience 4): Must have strong expertise in SQL, data warehousing, and database technologies, including experience with both SQL and NoSQL databases. - 6. Mandatory (Experience 5): Must have hands-on experience working with Big Data technologies and distributed data processing, including designing and optimizing large-scale data pipelines. - 7. Mandatory (Experience 6): Must have experience with data pipeline troubleshooting, monitoring, data quality, governance, and security best practices. - 8. Mandatory (Education): B.Tech / B.E./ M.Tech - 9. Mandatory (Notice Period): Immediate joiners or candidates who can join within 2 weeks. - 10. Preferred (Experience 1): Experience with Apache Airflow, Docker, and Kubernetes for pipeline orchestration and containerized data engineering workloads.