AI Engineer

Role Overview As an AI Engineer - RAG Platform Optimization within banks central AI capability, you take ownership of continuously improving the accuracy, retrieval quality, and response effectiveness of the bank's enterprise Retrieval-Augmented Generation (RAG) platform. Operating at the convergence of Information Retrieval, Data Science, and Software Engineering, your core mission is failure analysis, hypothesis-driven experimentation, and evaluation framework optimization. Rather than managing people or building standard web UI applications, you focus strictly on measurable model quality-analyzing production telemetry in Databricks, running chunking/embedding experiments, building continuous feedback loops, and ensuring every release raises the performance benchmark for thousands of internal users. Key Responsibilities: • Continuous RAG Model Optimization: Perform failure analysis on low-quality responses, identify underlying retrieval or grounding flaws, and run large-scale experiments on chunking, embedding models, prompt optimization, and reranking strategies. • GenAI Evaluation & Measurement Frameworks: Build and enhance automated evaluation pipelines to measure retrieval precision/recall, response correctness, hallucination rates, and user satisfaction metrics. • Data Analysis & Telemetry Insights: Analyze user interaction data, production telemetry, and evaluation datasets within Databricks to detect performance regressions and uncover quality improvement opportunities. • Production Pipeline Implementation: Translate data-driven experimentation results into scalable, maintainable Python production code and automated CI/CD data pipelines on Azure. • Cross-Functional Technical Alignment: Partner with Software Engineers, Data Scientists, Platform Engineers, and Product Managers to embed evaluation metrics into automated release pipelines. Your Deliverables: • Automated GenAI evaluation suites and synthetic benchmark datasets tracking retrieval and response quality. • Optimized chunking, embedding, reranking, and prompt strategies deployed directly into production pipelines. • Production telemetry dashboards in Databricks tracking model performance metrics, error categories, and user trust indicators. • CI/CD-integrated testing scripts that prevent quality regressions prior to production deployments. Your Profile: • Education (Hard Requirement): Master's degree (WO) or equivalent academic level in Data Science, Artificial Intelligence, Computer Science, Mathematics, Statistics, or a related quantitative field. • Experience & Track Record (Hard Requirements): • Minimum of 3+ years of professional experience in machine learning, applied AI, or data science. • Demonstrable track record of improving production-grade RAG systems or advanced retrieval-based AI solutions. • Strong hands-on proficiency in Python, Databricks, LLM architecture, and Azure cloud environments . • Proven experience designing evaluation methodologies, quality metrics, and continuous feedback loops for GenAI systems. • Proficiency in written and spoken English (corporate business language). • Nice-to-Have Technical Experience: • Experience with fine-tuning language models and reranking algorithms. • Hands-on experience with AI observability tools (e.g., MLflow, TruLens, Ragas, LangSmith). • Knowledge of synthetic evaluation dataset creation and benchmark curation.