SG Analytics - Data Engineering Manager

Roles and Responsibilities: - Build and evolve the EisnerAI lakehouse on ADLS Gen2 and Azure Databricks using a medallion architecture. - Develop batch and streaming ETL/ELT pipelines for firm source systems, client data, and third-party feeds. - Create reusable, configuration-driven ingestion frameworks to onboard new data sources rapidly. - Curate vector-ready data products to ground Retrieval-Augmented Generation (RAG) models via Azure AI Search. - Implement end-to-end data governance, access controls, and lineage via Unity Catalog and Microsoft Purview. - Optimize platform performance and manage cloud compute costs through partitioning and Z-ordering. - Establish CI/CD pipelines, automated data quality checks, and contract testing for pipeline reliability. Preferred Candidate Profile: - Experience: Minimum 4 years of hands-on experience building production data pipelines and platforms. - Education: Bachelor's degree in Computer Science, Engineering, Data Science, or a related technical field. - Technical Expertise: - Strong programming skills in Python, SQL, and PySpark at scale. - Proven delivery on the Azure data stack, including ADLS Gen2, Databricks, and Azure Data Factory. - Working knowledge of Delta Lake, columnar file formats (Parquet, Avro), and data governance. - Excellent communication skills to partner with cross-functional AI developers and product owners.