Cognida.ai - Senior Data & AI Engineer - Databricks

Position Overview : We are seeking a dynamic, self-driven Data & AI Engineer with deep expertise in the Databricks ecosystem to design, build, and accelerate our next-generation data architecture. In this role, you won't just write backend pipelines; you will be the bridge between complex data engineering and tangible business value. A critical measure of success in this role is the ability to rapidly build and execute concrete, high-impact technical demonstrations that showcase the power of unified data and AI. Because we operate globally, you must thrive in an autonomous, highly flexible environment, collaborating effectively across diverse time zones. Education : Bachelor's degree , BE/B. Tech in CS or equivalent Experience : 9 to 14 years Location : Hyderabad, India (Hybrid) Key Responsibilities : Databricks Architecture & Governance : - Design and implement robust data pipelines utilizing the Databricks Lakehouse platform. Take ownership of data governance, security, and fine-grained lineage using Unity Catalog. Rapid Prototyping & Demos : - Quickly conceptualize, build, and present functional, end-to-end data and AI demos (e.g., live dashboards, real-time pipelines, or LLM/RAG prototypes) to stakeholders and clients. Core Data Engineering : - Build scalable, efficient ETL/ELT pipelines using Spark (PySpark/SQL), Delta Lake, and structured streaming. Ensure optimal performance tuning, partitioning, and cost-management of Databricks clusters. Global Collaboration & Autonomy : - Work independently with minimal supervision, managing your own schedule to align with global, cross-functional teams. Required Skills & Qualifications : Databricks & Governance Mastery : - Proven track record implementing Databricks Unity Catalog for data governance, access control, and data lineage. - Deep understanding of Databricks architecture, including Delta Lake, serverless compute, and Workflows. Core Data Engineering Excellence : - Strong foundational knowledge of data engineering principles (star schemas, dimensional modeling, data warehousing vs. data lakes). - Advanced proficiency in Python/PySpark and SQL. - Experience building robust orchestrations and handling both batch and real-time streaming data. Agility & Communication : - Exceptional ability to build quick, concrete POCs (Proof of Concepts) and technical demos that clearly translate data capabilities into business solutions. - Excellent communication skills, with the ability to clearly articulate technical concepts to both technical and non-technical audiences. Preferred Qualifications : - Certifications such as Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional. - Experience integrating Databricks with enterprise ecosystems, cloud provider native services (AWS/Azure/GCP), or modern BI tools. - Exposure to building AI/ML workloads (e.g., MLflow, LLM integration, or RAG frameworks) within Databricks.