Senior Data Engineer

Role:- Senior Data Engineer Location:- Pune/Bangalore| Responsibilities: • Design and implement data pipelines to cleanse and transform data in preparation for feeding to AI/Forecasting, Price Optimization, and related Supply Chain SaaS softwareproducts. • Collaborate closely with Data Scientists to support feature engineering and operationalize AI/ML and mathematical optimization models. • Develop efficient and robust data transformation code in PySpark. • Build and maintain workflows to automate data pipelines using tools such as Databricks and Airflow. • Optimize performance of data pipelines to ensure minimal cost. • Develop data validation tests to assess the quality of the input data and recommend remediation measures. • Conduct performance testing and profiling of the code using a variety of tools and techniques. • Create and maintain comprehensive technical documentation for data pipelines and architecture. Qualifications: Job Requirements: • Minimum Education: Bachelor's degree in computer science, Engineering, or a related field (or equivalent work experience). • Minimum Work Experience (years): o 3 - 6 years of general Data Engineering experience. o Within that timeframe, experience is expected in: Python/PySpark, SQL, and relational or NoSQL databases. • Key Skills and Competencies: o Experience working with AWS, Azure, or GCP cloud environments. o Experience building end-to-end data pipelines using PySpark or SQL. Strong understanding of parallel processing, clusters and compute management in the cloud. o Experience with data orchestration tools like Databricks, Airflow (or similar tools like Snowflake, Dagster, etc.). o Excellent verbal and written communication skills, especially in technical communications. o Demonstrated ability to learn new technologies quickly and independently. o Preferred (but not required): Familiarity with Data Science concepts (e.g., Machine Learning) and a desire to grow into hybrid engineering/data science tasks. o Ability to work and achieve stretch goals in a very innovative and fast-paced environment. • Licenses/Certifications, special qualifications: N/A Skills: snowflake , Artificial Intelligence, Pyspark, Data Engineer, Azure Data Bricks, Airflow, AI ML, Databricks, Python Experience: 3.00-9.00 Years