Production AI Engineer - Vice President

The Production Engineer is a pivotal role within Citi's Technology organisation, responsible for designing, building, and operating the intelligent systems that underpin our global production environment.

This is an engineering-first position at the intersection of software craftsmanship, AI-native development, and large-scale distributed systems.

As part of a multi-year transformation journey, the successful candidate will help define what production engineering looks like in an era of autonomous agentsa, generative AI, and self-ahealing infrastructure.

You will be expected to write production-grade code daily, design agentic workflows, and contribute meaningfully to the evolution of our AI engineering practices across Citi's technology hub.

The role requires a comprehensive understanding of multiple areas within a function and how they interact to achieve the objectives of the function. Applies in-depth understanding of the business impact of technical contributions. Accountable for delivery of a full range of end-to-end projects.

Excellent communication skills required to negotiate internally. Involved in short- to medium-term planning of actions and resources for own area.

**Responsibilities:** + Designs, develops, and maintains production-grade software systems with a strong emphasis on reliability, scalability, and operational excellence across Citi's global technology estate.

+ Architects and implements **agentic AI workflows** - building autonomous systems that can reason, plan, and act across production environments with minimal human intervention.

+ Applies advanced **prompt engineering techniques** to integrate large language models (LLMs) into operational tooling, incident response pipelines, and developer productivity platforms.

+ Leads the development of **AI-native observability** solutions - leveraging intelligent agents to detect anomalies, predict failures, and automate remediation before issues impact end users.

+ Writes clean, well-tested, and well-documented code across the full stack; champions engineering best practices including code review, pair programming, and test-driven development.

+ Drives **Continuous Delivery and Automation** efforts across supported applications by means of Root Cause Analysis reviews, knowledge management, performance tuning, and user training.

+ Operates and evolves CI/CD pipelines, Infrastructure-as-Code tooling, and GitOps workflows to support rapid, safe delivery of software at scale. + Collaborates with platform, data, and product engineering teams to embed AI capabilities into the production lifecycle - from deployment to decommission.

+ Implements the **Agile Framework** through one of its implementations (SCRUM or Kanban) and ensures it integrates with overall organisation processes.

+ Operates within a highly regulated financial environment, maintaining in-depth understanding of compliance requirements and their implications for system design and data handling.

+ Coaches and mentors team members on AI engineering practices, prompt design patterns, and agentic system architecture - fostering a culture of continuous learning and technical excellence.

+ Avidly communicates progress and project status across the organisation and ensures that stakeholders are managed appropriately throughout the execution period. + Fosters a culture that promotes transparency and innovation for increased team productivity.

**Qualifications** + Demonstrable experience in a critical **software engineering or production engineering role** with high business impact and a strong programming foundation (Java, Python, Go, or equivalent).

+ Hands-on experience with **AI/ML engineering** - including working with LLM APIs (OpenAI, Anthropic, Gemini, or open-source equivalents), embedding models, and vector databases.

+ Proven expertise in **prompt engineering** : designing, iterating, and evaluating prompts for production use cases including classification, summarisation, code generation, and autonomous decision-making.

+ Experience designing and deploying **agentic systems** using frameworks such as LangChain, LangGraph, AutoGen, CrewAI, or equivalent - including multi-agent orchestration and tool-use patterns.

+ Excellent engineering skills and strong understanding of **Software Development Lifecycle** , GitOps, and modern DevSecOps practices. + Excellent working knowledge of key computer science concepts (networking, operating systems, virtualisation, containerisation, etc.)

+ Polyglot full-stack developer mentality and ability to pick up new languages and skills. + Excellent debugging and analytical skills: ability to isolate root cause across networking/infrastructure, application, and database stacks.

+ Operational experience of deploying and running services at scale on top of **Docker/Kubernetes** stack and a service mesh (Istio or equivalent) is highly desirable. + Operational experience with orchestration tools for **CI/CD and Infrastructure-as-Code** tooling (Terraform, CloudFormation, Pulumi, etc.)

is highly desirable. + Experience of delivering software using **Agile delivery methodologies** is a must (SCRUM/Kanban). + Operational experience of using **middleware technologies** (MQ, Apache Kafka, etc.) to run services at scale is desirable.

+ Strong experience with **end-to-end observability stacks** (Datadog, AppDynamics, Dynatrace, etc.) is desirable. + Degree in Computer Science, Mathematics, Physics, or a related technical subject is desirable. + Experience of senior stakeholder management.

+ Consistently demonstrates clear and concise written and verbal communication skills. + Ability to operate in a global environment with on-/near-/off-shore matrix reporting structures.

Beyond technical capability, the Production Engineer who will thrive in this role brings a distinct set of human qualities that amplify their engineering impact and elevate those around them. + **Learnability** - Rapidly acquires new skills, frameworks, and paradigms.

In a field evolving as fast as AI engineering, the ability to learn is the most durable skill of all. + **Teachability** - Receives feedback with openness and intellectual humility. Actively seeks mentorship and applies guidance to accelerate growth. + **Flexibility & Adaptability** - Thrives in ambiguity.

Pivots gracefully when requirements shift, technology evolves, or priorities change - without losing momentum or quality. + **Engineering Mindset** - Approaches every problem systematically: decomposing complexity, forming hypotheses, and validating solutions with rigour and precision.

+ **Product-Minded Thinking** - Understands that code serves users and business outcomes. Balances technical elegance with pragmatic delivery and user impact. + **Collaborative Spirit** - Builds trust across disciplines - engineering, product, operations, and leadership.

Elevates the team's collective output through generosity and clear communication. + **Intellectual Curiosity** - Asks "why" before "how". Explores the edges of what's possible with AI and production systems, driven by genuine fascination rather than obligation.

+ **Ownership & Accountability** - Takes end-to-end responsibility for what they build. Does not hand off problems - follows through from design to deployment to post-incident review.

**What we'll provide you:** By joining Citi, you will not only be part of a business casual workplace with a hybrid working model (up to 2 days working at home per week), but also receive a competitive base salary (which is annually reviewed), and enjoy a whole host of additional benefits such as: + 27 days annual leave (plus bank holidays) + A discretional annual performance related bonus + Private Medical Care & Life Insurance + Employee Assistance Program + Pension Plan + Paid Parental Leave + Special discounts for employees, family, and friends + Access to an array of learning and development resources Alongside these benefits Citi is committed to ensuring our workplace is where