Azure Data Engineer

Job Title: Azure Data Engineer Location: Greenfield, IN (Onsite) Employment Type: Full-Time (FTE) Role Summary We are looking for an experienced Data Engineering Tech Lead with strong expertise in Azure Databricks, PySpark, Azure Data Factory (ADF), ADLS Gen2, Azure SQL, Python, and Azure Cloud. The role will lead the design, development, optimization, and maintenance of scalable enterprise data engineering and Lakehouse solutions, including batch and near-real-time data processing. Technical Skills and Experience • 4+ years of experience in Azure Databricks with PySpark. • 2+ years of experience in Databricks Workflows and Unity Catalog. • 3+ years of experience in Azure Data Factory (ADF). • 3+ years of experience in ADLS Gen2. • 3+ years of experience in Azure SQL. • 5+ years of experience in the Azure Cloud platform. • 2+ years of experience in Python programming and package builds. Key Responsibilities Data Engineering and Lakehouse Solutions • Handle analytics workloads covering the design, development, and maintenance of Lakehouse solutions. • Source data from platforms such as ERP systems, APIs, relational databases, NoSQL databases, and on-premises sources using Databricks and PySpark. • Support batch and near-real-time data ingestion, transformation, and processing. • Implement scalable and efficient data processing solutions for enterprise data platforms. Azure Databricks and PySpark • Optimize Spark jobs and manage large-scale data processing using RDD and DataFrame APIs. • Apply partitioning strategies, file format optimization using Parquet and Delta, and Spark SQL tuning. • Work with Databricks runtime versions, cluster policies, libraries, and workspace management. Unity Catalog and Data Governance • Govern and manage data access for the Azure Data Lakehouse using Unity Catalog. • Configure data permissions, object lineage, and access policies. • Understand integration with Azure AD, external metastores, and audit trails. Azure Data Factory and Workflow Orchestration • Build efficient orchestration solutions using Azure Data Factory and Databricks Workflows. • Design modular, reusable workflows using tasks, triggers, and dependencies. • Implement dynamic expressions, parameterized pipelines, custom activities, and triggers. • Configure integration runtimes and optimize pipeline performance and error-handling strategies. ADLS Gen2 and Data Storage • Implement secure, hierarchical namespace-based data lake storage for structured and semi-structured data. • Develop Bronze, Silver, and Gold data layers using ADLS Gen2. • Manage lifecycle policies, role-based access control (RBAC), access control lists (ACLs), and folder-level security. • Apply best practices for file partitioning, retention management, and storage performance optimization. Azure SQL and Python Development • Develop T-SQL queries and stored procedures and manage metadata layers on Azure SQL. • Write modular, testable Python code for data transformations, utility functions, and reusable components. • Manage Python environments, dependencies, and package libraries using tools such as pip, Poetry, or Conda. • Write unit tests using PyTest or unittest and integrate testing into CI/CD pipelines. Azure Cloud and DevOps • Work across the Azure ecosystem, including networking, security, monitoring, and cost management for data engineering workloads. • Understand VNets, Private Endpoints, Key Vaults, Managed Identities, and Azure Monitor. • Use DevOps tools for deployment automation, including Azure DevOps, ARM, Bicep, and Terraform. • Follow version control workflows using Git and apply code quality tools such as SonarQube and pylint. Technical Leadership and Documentation • Lead solution design discussions and mentor junior engineers. • Ensure adherence to coding guidelines, design patterns, and peer review processes. • Prepare technical design documents, High-Level Designs (HLDs), Low-Level Designs (LLDs), and architecture diagrams. • Guide the team technically and ensure alignment with established coding principles and standards. Stakeholder Collaboration • Communicate technical concepts clearly to technical and business audiences. • Engage proactively with internal teams and customer stakeholders. • Collaborate with product owners, QA teams, and business analysts to translate requirements into deliverables. • Provide regular work updates and contribute to Agile/Scrum projects using tools such as Jira and Azure DevOps. Soft Skills • Strong verbal and written communication skills. • Strong interpersonal skills and the ability to build productive relationships with team members. • Ability to provide constructive feedback during code reviews and receive feedback. • Strong problem-solving and analytical thinking skills. • Ability to troubleshoot and resolve technical issues efficiently. • Proactive, collaborative, and diligent approach to responsibilities. • Experience working in Agile/Scrum environments. Good-to-Have Skills • Azure Entra ID / Active Directory and GitHub Actions. • Orchestration experience using Airflow, Dagster, or Logic Apps. • Event-driven architecture experience using Kafka or Azure Event Hubs. • Exposure to Google Cloud Pub/Sub. • Change Data Capture (CDC) experience, preferably using Debezium. • Experience with data migration projects involving Azure Synapse and Databricks Lakehouse. • Hands-on experience managing Azure Data Lake Storage solutions. • Google Cloud Storage experience.