Principal Software Engineer

Make Your Mark: As a Principal Software Engineer, you will play a crucial role in delivering high quality releases to our customers by defining application architecture, designing, developing, troubleshooting, maintaining, optimizing, and scaling BlackLine's cloud-based backend services and driving innovation into our product suite. In this role, you will work with management to identify strategic directions for system architecture. You will research, test, benchmark and evaluate new technologies, propose scaling solutions, collaborate with other teams to define, and build new features, optimize existing features while writing elegant code, coaching, and mentoring less experienced software engineers and communicating with people at all levels. In this position, every bit of your software design skill be drawn upon in order to drive towards and implement best of breed features in a rapid iterative environment. Your attention to detail, tenacity, and creative problem-solving and thinking ahead will directly impact the company's success, as well as the success of your peers across the organization. If you have a passion for delivering products that make a difference, are driven to make things better in all that you do and are eager to work in a collaborative agile environment where Developers are dedicated to solving customer problems and innovating, you will be a great fit for the team. You'll Get To: We are looking for an accomplished, hands-on Principal Engineer with 18+ years of software engineering experience and a strong record of setting technical direction in Agentic AI, Generative AI systems and Software/Data engineering . This is the most senior individual contributor role on the Agentic AI platform. You will set the multi-year technical vision and own the architecture across products, while staying close enough to the code to make the critical calls. You will guide Senior Staff Engineers, Staff Engineers and Technical Leads across several teams. The ideal candidate has led 5+ enterprise-scale AI programs from strategy and solution design through to production at scale, and has shaped platform architecture that several product teams build on. You can define an organization-wide AI architecture strategy and present it to executives, and you can also prototype and write production-grade Python when a problem needs it. You bring the vision to set direction across teams, the judgment to make high-stakes architecture and investment decisions, and the credibility to raise the engineering bar across the company. The role spans strategy, architecture and execution. You will set the AI platform roadmap, define the governance and trust frameworks that make agents enterprise-ready on a secure, compliant financial platform, and establish the engineering standards that every AI product team follows. Key Responsibilities Technical Strategy & Vision • Define and own the multi-year technical strategy and architecture roadmap for the Agentic AI platform across all product lines. • Make build-vs-buy, technology selection and platform investment decisions, and present the trade-offs, risks and ROI to executive leadership. • Track the agentic AI landscape: frameworks, protocols, models, regulation. Turn those shifts into strategic bets and platform direction. • Identify new opportunities where AI can deliver business value across product areas, and shape them into fundable programs. Architecture Leadership • Own the end-to-end architecture of the Agentic AI system: agent registry, policy engine, metrics ingestion pipeline, anomaly detection and HITL orchestration. • Define the reference architecture for multi-agent systems: trust tier boundaries (T0–T3), separation of concerns, fault isolation and full observability. • Set organization-wide patterns for data flows, integrations (REST, gRPC, SSE, event-driven) and storage strategies for AI workloads in a multi-tenant FinTech environment. • Own the design principles for durable DAG execution, pause and resume of long-running financial close workflows, event-sourced audit trails and SLA-tracked HITL approval gates. • Run the architecture governance process: ADRs, cross-team design reviews and production readiness assessments for critical systems. Technical Leadership & Organizational Influence • Split the platform into programs and workstreams. Assign ownership to Senior Staff and Staff Engineers, with clear API contracts, interface definitions and success metrics. • Act as the senior technical authority for Engineering, Product, Security, Compliance, Finance and Data Platform leaders, and align architecture decisions across organizations. • Represent the AI Platform in executive and cross-functional technical forums, and externally at industry events, through publications and in partner discussions. • Find and resolve cross-team technical dependencies and systemic risks early in planning. Mentorship & Talent Development • Grow Senior Staff Engineers, Staff Engineers and Technical Leads into the next generation of technical leaders through architecture working sessions, design clinics and 1:1 coaching. • Review designs from senior engineers. Find gaps in governance, observability or compliance thinking and guide them to better solutions rather than overriding them. • Shape technical hiring standards and help attract top senior talent to the organization. Engineering Standards & Governance • Define and drive adoption of engineering standards across all AI product teams: agent identity patterns, audit trail requirements, HITL design principles, Kafka topic design and eval harness setup. • Set the quality bars: performance benchmarks, eval thresholds, observability standards and security posture. • Own the agent governance and certification framework, including the T1 agent certification pipeline and its evaluation harnesses for model quality, factuality and safety at scale. • Drive the production rollouts and release readiness for all releases • Support the on-call and be the first point of contact for any incidents reported internally or from customers Hands-on Technical Depth • Step in directly on the hardest problems with code, prototypes, design sketches or paired debugging: novel system design, complex integrations, critical production incidents. This is not a review-only role. • Guide the design and delivery of intelligent agents for complex, multi-step financial reasoning, using ReAct, Plan-and-Execute and other advanced reasoning strategies. • Set the orchestration strategy across LangGraph, Temporal, Google Workflows or equivalent, with documented architectural reasoning. • Define deployment architecture on GCP (Vertex AI Agent Builder, Cloud Run, GKE) or AWS/Azure equivalents, including Kubernetes namespace isolation for T1 certified partner agents. • Shape MCP (Model Context Protocol) server and client patterns for external agent integration (T2/T3 trust tiers via SSE/HTTPS). • Guide RAG architecture with vector stores (Pinecone, Weaviate, pgvector) for retrieval over domain-specific financial documents. • Lead model strategy across foundation models (Gemini, GPT-4, Claude), covering fine-tuning, prompt engineering and evaluation for accounting tasks such as close commentary and variance explanation. • Own the inference cost strategy: caching, batching, model routing and quantization. Keep token costs visible in Control Tower. What You'll Bring: Required • 18+ years of software engineering experience, including significant time as a senior or principal-level technical leader. • A record of setting technical strategy and architecture across multiple teams or product lines. • Led 5+ enterprise-scale AI/ML or GenAI programs from strategy through to production at scale. • Deep expertise in agentic AI architectures, multi-agent orchestration, LLM evaluation and AI governance. • Strong hands-on proficiency in Python and distributed systems design. • Extensive expe