Principal Engineer - Duo Agent Platform

About the role

We are looking for a Principal Engineer to help build GitLab Duo Agent Platform, the agentic orchestration layer that runs AI agents and flows across the entire software lifecycle.

Principal Engineers at GitLab are recognized experts across multiple technology domains and represent the senior technical leadership within their organization. In this role you will set technical direction for a significant part of the Agent Platform, working across several teams to turn agentic capabilities into something engineering organizations can actually depend on: predictable, observable, governable, and fast enough to sit in the critical path of real development work.

The Agent Platform is not a single service. It spans agent execution and orchestration, flow definition and event-driven triggering, the AI Catalog through which customers discover and manage agents, model routing and evaluation, and the governance surface that decides which agents may act where and under whose identity. It runs on GitLab.com, GitLab Dedicated, and Self-Managed installations, including deployments that use self-hosted models to meet compliance requirements. You will work on the parts of that system where the hard architectural questions live, and you will do it hands-on.

This role reports into the VP of the AI Engineering organization and works closely with Engineering Managers, Staff Engineers, Product Management, Infrastructure and Security.

What you'll do

• Define the technical direction for a significant area of Duo Agent Platform across several teams, and partner with Engineering Managers and Product to turn it into a roadmap with a six-month horizon.
• Build the orchestration layer that makes agents and flows reliable: session lifecycle, tool invocation, context assembly, concurrency, retries, timeouts, and failure modes that degrade gracefully instead of stalling a developer's workflow.
• Hands on work on driving topics throughout the whole organisation from POC level, architecture to full implementation. 
• Participate in Incident Management on-call rotation, to help meet availability goals for GitLab.com.
• Strong product builder and ownership mentality. Understanding of product strategy, technical strategy, delivering customer value and iteration towards a successful product
• Write clear, opinionated design documents, architecture narratives, and decision records that let other teams make aligned decisions. Ensure the architecture holds across multi-tenant GitLab.com, Dedicated, and Self-Managed deployments, including self-hosted model configurations, and respects data governance boundaries in each.
• Use the platform on our own work, find where it falls short of what we claim, and feed that back through improvements into the product.
• Mentor Staff and Senior Engineers, and raise the bar for experimentation rigor and cross-team collaboration.
• Work across the whole platform on different projects, features and teams. This reaches across:
• Design the extensibility surface so customers can build custom agents and flows, connect their own tools and internal systems, and bring external agents such as Claude Code and Codex into GitLab workflows without weakening the platform's guarantees.
• Drive down the cost and latency of agentic requests, and work with Product on how model selection, routing, and credit consumption behave as usage grows.
• Define the evaluation practice for the platform: offline benchmarks and online experiments that measure task completion, correctness, tool-call accuracy, latency, and cost per task, and that catch regressions before customers do.
• Build the governance and safety surface with Security and Compliance: policy-driven controls over which agents and models may run where, permission and identity semantics that align to existing roles and group structure, human-in-the-loop checkpoints, traceability of every flow session, and audit that stands up to enterprise review.
• Complete Interview Training and participate in hiring for the department as a technical interviewer or panel member.
• Take part in on-call or operational support where your team’s ownership model requires it.

What you'll bring

• 8+ years of software engineering experience, including 3+ years at Staff, Principal level or equivalent senior technical leadership.
• Hands-on production experience building on large language models: agent loops, tool calling, context management, retrieval, prompt and model evaluation. Experience shipping this to real users, not only prototyping it.
• Strong background in distributed systems, including service decomposition, fault tolerance, observability, and operational resilience under multi-tenant load.
• A track record of defining evaluation frameworks and using benchmark and experiment results to make architecture decisions, rather than shipping on intuition.
• Experience designing human-in-the-loop controls, safety guardrails, and responsible AI practices for systems that act on a customer's behalf.
• Ability to drive alignment across Engineering, Product, Infrastructure, and Security on problems that start out ambiguous.
• Experience mentoring senior engineers and shaping technical direction across multiple teams without direct authority.
• Comfort working in one or more of Ruby, Go, Python, or TypeScript, and willingness to work across the stack the platform spans.
• Excellent written and asynchronous communication, and the ability to work effectively in a fully remote, globally distributed organization.

About the team

The AI engineering organisation is building the product and infrastructure that make GitLab an effective environment for running agents across the software development lifecycle. We’re distributed across the world and partner with teams throughout GitLab to help the GitLab Duo Agent Platform become the backbone of GitLab's AI features. You will report directly to the VP of Engineering for AI and especially work together with all the teams, the engineering management and other principal engineers in the AI department. This is an opportunity to lead a consequential group as AI changes how software is built.

United States Salary Range 250000 - 325000 USD