eClerx - Data Engineer - Google Cloud Platform

Role : GCP Data Engineer Experience : Not specified Location : Navi Mumbai / Pune Employment Type : Full-time, Permanent Industry : IT Services & Consulting Department : Engineering - Software & QA Role Category : Software Development About the Role : We are looking for a skilled GCP Data Engineer to design, develop, and maintain scalable data pipelines and ETL/ELT frameworks on Google Cloud Platform (GCP) . The role will involve working with large-scale campaign and clickstream datasets , building reliable data processing workflows, optimizing BigQuery performance and costs, and implementing automation, monitoring, and alerting for critical data pipelines. The ideal candidate should have strong hands-on expertise in GCP, BigQuery, SQL, Python/PySpark, and Airflow/Composer , along with an understanding of CI/CD and Git-based development workflows. Key Responsibilities : - Design, develop, and implement scalable data pipelines on GCP using services such as BigQuery, Cloud Storage, and Airflow/Composer. - Build robust ETL/ELT pipelines to process and transform large volumes of campaign, clickstream, and other business data. - Develop efficient data ingestion and transformation workflows to support analytics and downstream applications. - Write optimized SQL queries in BigQuery for large-scale data processing and analytics. - Optimize BigQuery query performance and processing costs through effective data modelling, partitioning, clustering, query optimization, and other best practices. - Develop reusable and scalable data engineering frameworks for batch and scheduled data processing. - Automate data workflows and pipeline execution using Airflow/Cloud Composer . - Implement appropriate monitoring, logging, alerting, and failure-handling mechanisms for critical data pipelines. - Identify and troubleshoot pipeline failures, data quality issues, performance bottlenecks, and integration-related problems. - Build and maintain reliable data processing workflows with appropriate data validation and quality checks. - Collaborate with analytics, product, engineering, and business teams to understand data requirements and deliver scalable solutions. - Follow best practices for version control, code quality, testing, deployment, and documentation . - Support CI/CD processes for data engineering workflows using tools such as Jenkins or equivalent platforms . - Work with Git-based development workflows for managing and deploying data engineering code. - Continuously identify opportunities to improve pipeline reliability, scalability, performance, and operational efficiency. Required Skills & Experience : - Strong hands-on experience in Data Engineering and building scalable data pipelines. - Strong proficiency in SQL , preferably with extensive experience in Google BigQuery . - Strong programming skills in Python and/or PySpark . - Hands-on experience with Google Cloud Platform (GCP) . - Strong experience with BigQuery for data storage, transformation, querying, and optimization. - Experience working with Google Cloud Storage (GCS) . - Experience developing ETL/ELT frameworks for large-scale datasets. - Hands-on experience with Apache Airflow or Google Cloud Composer for workflow orchestration. - Understanding of data pipeline monitoring, logging, alerting, and troubleshooting. - Familiarity with CI/CD practices and tools such as Jenkins. - Good understanding of Git and version-control workflows . - Strong analytical and problem-solving skills. Good to Have : - Experience working with campaign, marketing, customer, or clickstream data . - Experience handling large-volume datasets on cloud platforms. - Knowledge of BigQuery partitioning and clustering . - Experience with data quality frameworks and validation processes. - Exposure to other GCP services used in data engineering. - Experience working in Agile development environments