Qentelli Solutions - Data Engineer - SQL/Python

Job Summary: We are looking for an experienced Data Engineer with expertise in AWS, SQL, and Python to help design, implement, and optimize our data pipelines and data processing workflows. The ideal candidate will have hands-on experience working with large datasets in the cloud and the ability to work with cutting-edge technologies to extract actionable insights from data. You will be responsible for developing and managing scalable data pipelines, working with data scientists and analysts, and ensuring the availability and performance of data systems. Key Responsibilities: - Design, develop, and maintain scalable data pipelines and ETL processes using AWS technologies (e.g., AWS S3, AWS Glue, Redshift, Lambda, etc.). - Implement batch and real-time data processing workflows. - Write complex SQL queries to extract, transform, and load data from various relational and non-relational data sources. - Collaborate with data scientists, analysts, and other engineering teams to define data requirements and ensure the availability and quality of data for analytics. - Develop and manage automated processes for data integration, cleansing, and transformation. - Work with large-scale datasets and optimize data processing performance for speed and scalability. - Monitor data pipeline performance, troubleshoot and resolve issues, and continuously optimize for efficiency and cost. - Implement best practices for data governance, security, and compliance in the cloud. - Participate in code reviews, maintain proper documentation, and ensure the use of best practices for code quality and reliability. - Stay up-to-date with emerging cloud technologies and tools, and make recommendations for improvements to the data pipeline architecture. Required Skills and Experience: - 8+ years of experience as a Data Engineer or similar role, with a focus on cloud technologies and large-scale data processing. - Strong experience with AWS services (e.g., AWS S3, AWS Glue, Redshift, Lambda, Kinesis, etc.). - Expertise in Python for building data pipelines and writing automation scripts. - Solid understanding of SQL and experience working with relational databases such as MySQL, PostgreSQL, or Redshift. - Hands-on experience of AWS Glue. - Experience working with data lake architectures, data warehousing solutions, and building end-to-end ETL pipelines. - Familiarity with data modeling and understanding of the principles behind data normalization, denormalization, and star schemas. - Knowledge of data versioning, logging, and error handling in cloud-based systems. - Strong problem-solving skills and the ability to work with large datasets and complex data processing tasks. - Familiarity with DevOps practices and tools (e.g., Git, CI/CD pipelines). - Experience working with AWS Cost Management to ensure cost-efficient solutions for data processing. - Excellent communication skills and the ability to work collaboratively in a team environment.