Python, AWS Services, Pyspark Developer

CGI is seeking an experienced Python, AWS Services, Pyspark Developer to design, develop, optimize, and maintain scalable data engineering solutions on cloud platforms. The ideal candidate will have strong hands-on experience in Python programming, distributed data processing using PySpark, and building data pipelines using AWS services. The candidate will be responsible for developing robust ETL/ELT pipelines, processing large volumes of structured and unstructured data, implementing data transformations, and ensuring data quality, reliability, and performance. The role requires close collaboration with Data Engineers, Data Architects, Business Analysts, and cross-functional teams to deliver enterprise-grade data solutions. Your future duties and responsibilities • Design, develop, test, and maintain scalable data processing applications using Python and PySpark. • Build and maintain ETL/ELT pipelines for data ingestion, transformation, validation, and integration. • Develop distributed data processing solutions using Apache Spark and PySpark to process large datasets efficiently. • Design, implement, and manage cloud-based data pipelines using AWS services. • Work with AWS services such as Amazon S3, AWS Glue, AWS Lambda, Amazon EMR, Amazon Redshift, AWS Step Functions, and Amazon CloudWatch, as applicable to the project. • Develop data extraction, transformation, and loading processes from multiple data sources into data lakes and data warehouses. • Implement data transformation logic, business rules, aggregations, joins, and data cleansing using Python and PySpark. • Optimize Spark jobs through partitioning, caching, join strategies, and resource tuning to improve performance and reduce processing time. • Develop and maintain reusable Python modules, libraries, utilities, and automation scripts. • Implement data validation, reconciliation, error handling, logging, and exception management within data pipelines. • Monitor pipeline execution, troubleshoot failures, and resolve performance bottlenecks in production environments. • Work with relational databases and data platforms using SQL for data extraction, analysis, validation, and reconciliation. • Implement secure data processing practices using appropriate AWS IAM roles, policies, encryption, and access controls. • Support CI/CD pipelines, automated deployments, version control, and release management using tools such as Git and AWS-native or other DevOps solutions. • Collaborate with Data Architects, Data Engineers, DevOps teams, and business stakeholders to translate requirements into technical solutions. • Participate in code reviews, unit testing, integration testing, production deployments, and technical documentation. • Perform root cause analysis and implement corrective actions for data pipeline failures and production incidents. • Ensure adherence to coding standards, data governance requirements, security policies, and enterprise development practices. Required Qualifications To Be Successful In This Role • Design, develop, test, and maintain scalable data processing applications using Python and PySpark. • Build and maintain ETL/ELT pipelines for data ingestion, transformation, validation, and integration. • Develop distributed data processing solutions using Apache Spark and PySpark to process large datasets efficiently. • Design, implement, and manage cloud-based data pipelines using AWS services. • Work with AWS services such as Amazon S3, AWS Glue, AWS Lambda, Amazon EMR, Amazon Redshift, AWS Step Functions, and Amazon CloudWatch, as applicable to the project. • Develop data extraction, transformation, and loading processes from multiple data sources into data lakes and data warehouses. • Implement data transformation logic, business rules, aggregations, joins, and data cleansing using Python and PySpark. • Optimize Spark jobs through partitioning, caching, join strategies, and resource tuning to improve performance and reduce processing time. • Develop and maintain reusable Python modules, libraries, utilities, and automation scripts. • Implement data validation, reconciliation, error handling, logging, and exception management within data pipelines. • Monitor pipeline execution, troubleshoot failures, and resolve performance bottlenecks in production environments. • Work with relational databases and data platforms using SQL for data extraction, analysis, validation, and reconciliation. • Implement secure data processing practices using appropriate AWS IAM roles, policies, encryption, and access controls. • Support CI/CD pipelines, automated deployments, version control, and release management using tools such as Git and AWS-native or other DevOps solutions. • Collaborate with Data Architects, Data Engineers, DevOps teams, and business stakeholders to translate requirements into technical solutions. • Participate in code reviews, unit testing, integration testing, production deployments, and technical documentation. • Perform root cause analysis and implement corrective actions for data pipeline failures and production incidents. • Ensure adherence to coding standards, data governance requirements, security policies, and enterprise development practices. Together, as owners, let’s turn meaningful insights into action. Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because… You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction. Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise. You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons. That same commitment to fairness extends to how we use technology. To support our recruitment team, AI tools may be used to help assess applications though they never replace human judgement. All hiring decisions remain entirely in the hands of our recruitment professionals. Come join our team—one of the largest IT and business consulting services firms in the world.