Software Engineer - Python | SQL | Databricks | Dbt

Sydney Contract Engagement We are seeking a skilled Data Engineer with strong Databricks experience to join our growing data team. In this role, you will design, build, and optimize large-scale data pipelines, ensure high data quality, and support analytics a initiatives. Responsibilities • Develop and maintain scalable ETL/ELT pipelines using Databricks (Spark, Delta Lake). • Build and optimize data lakehouse architectures for batch and streaming workloads. • Collaborate with data analysts, data scientists, and product teams to deliver reliable datasets for reporting and ML models. • Implement best practices around data quality, observability, governance, and documentation. • Manage and optimize Databricks clusters , workflows, and job performance. • Integrate data from various sources (APIs, databases, cloud storage, SaaS platforms). • Support infrastructure using Azure/AWS and CI/CD for data pipelines. • Ensure compliance with data privacy and security standards. Required Qualifications • 4+ years of experience as a Data Engineer . • Hands-on experience with Databricks , Apache Spark , and Delta Lake . • Strong SQL and Python skills. • Experience building ETL/ELT pipelines at scale. • Familiarity with cloud platforms ( Azure, AWS, or GCP ). • Experience with Azure Databricks or AWS Databricks . • Understanding of data modeling, distributed computing, and data warehousing concepts. • Experience with data orchestration tools (Airflow, dbt, ADF, Prefect, etc.). • Experience with version control (Git) and CI/CD tools.