Ganit - Senior Data Engineer - Python/Spark

Senior Data Engineer About Ganit At Ganit, we are not just data geeks - we are a family of scientists, refiners, and artisans. Our mission is to bridge the gap between intelligence and action, translating data into valuable insights and strategies. We work with businesses to create state-of-the-art solutions that push the boundaries of data science. Roles & Responsibilities: - Data Engineering Excellence: Design and implement data pipelines using formats like JSON, Parquet, CSV, and ORC, utilizing batch and streaming ingestion. - Cloud Data Migration Leadership: Lead cloud migration projects, developing scalable Spark pipelines. - Medallion Architecture: Implement Bronze, Silver, and gold tables for scalable data systems. - Spark Code Optimization: Optimize Spark code to ensure efficient cloud migration. - Data Modeling: Develop and maintain data models with strong governance practices. - Data Cataloging & Quality: Implement cataloging strategies with Unity Catalog to maintain high-quality data. - Delta Live Table Leadership: Lead the design and implementation of Delta Live Tables (DLT) pipelines for secure, tamper-resistant data management. - Customer Collaboration: Collaborate with clients to optimize cloud migrations and ensure best practices in design and governance. Educational Qualifications: - Experience: Minimum 4 years of hands-on experience in data engineering, with a proven track record in complex pipeline development and cloud-based data migration projects. - Education: Bachelors or higher degree in Computer Science, Data Engineering, or a related field. Skills: - Proficiency in Spark, SQL, Python, and other relevant data processing technologies. - Expertise in on-premises to cloud Spark code optimization and Medallion Architecture. - Strong knowledge of Databricks and its components, including Delta Live Table (DLT) pipeline implementations. - Familiarity with AWS services (experience with additional cloud platforms like GCP or Azure is a plus). Soft Skills: - Excellent communication and collaboration skills, with the ability to work effectively with clients and internal teams. Certifications: - AWS/GCP/Azure Data Engineer Certification.