Data Engineer

RoleOverview Weare looking for a hands-on Data Engineer to design, build, and optimizescalable data pipelines on the Databricks Lakehouse platform. The role involvescollaborating with analysts, data scientists, and business teams to deliverreliable, analytics-ready data assets. KeyResponsibilities • Build and maintain batch and streaming data pipelines using PySpark and Databricks . • Implement medallion architecture (Bronze, Silver, and Gold layers). • Optimize Spark jobs for performance and cost efficiency. • Ensure data quality, lineage, and governance using Unity Catalog and automated validation. • Integrate data from APIs, databases, files, and event streams. • Implement CI/CD, version control, and workflow orchestration. • Monitor, troubleshoot, and document data pipelines and data models. • Collaborate with stakeholders to translate business requirements into technical solutions. RequiredQualifications • 3-5 years of hands-on data engineering experience. • Strong PySpark skills, including DataFrames, Spark SQL, UDFs, and performance tuning . • Proven experience building production pipelines on Databricks. • Working knowledge of Delta Lake, including ACID transactions, time travel, merge/upsert, and schema evolution . • Advanced SQL and strong Python programming skills . • Experience with Azure, AWS, or GCP cloud platforms. • Understanding of data modelling, Lakehouse architecture, and ETL/ELT patterns. • Familiarity with Git and CI/CD practices. PreferredQualifications • Databricks Data Engineer Associate or Professional certification. • Experience with Unity Catalog, Auto Loader, and Structured Streaming. • Familiarity with Airflow, Azure Data Factory, or Databricks Workflows. • Knowledge of Terraform and testing frameworks such as pytest. • Exposure to Kafka, Event Hubs, dbt, or MLOps workflows. • Experience working in regulated or enterprise environments. KeySkills • PySpark and Python • Databricks and Delta Lake • Advanced SQL • Data Pipeline Development • Medallion Architecture • Cloud Data Engineering • Data Quality and Governance • CI/CD and Git • Spark Performance Tuning DesiredAttributes • Strong problem-solving and debugging skills. EA Number: 11C4879 #J-18808-Ljbffr