All remote jobs

Principal Engineer, Duo Agent Platform

Review the role, location requirements, compensation details, and application process before deciding whether this opportunity fits your next career move.

Remote from
UK, USA, Canada+1 more, Poland
Salary
USD 250k–325k / yr
Employment
Full Time
Experience
Director
Published
Apply before
28 Nov 2026
Listing views
23
Application actions
2
Application toolkit

Make your next move.

Prepare your resume, explore your fit, and draft a cover letter for this opportunity.

AI Summary

The role, at a glance.

GitLab is seeking a Principal Engineer to set technical direction for the Duo Agent Platform, an AI-agent orchestration layer spanning execution, workflows, model routing, evaluation, governance, and extensibility. The role combines hands-on architecture and implementation with cross-team technical leadership across Engineering, Product, Infrastructure, and Security. The engineer will design resilient multi-tenant distributed systems, establish AI evaluation practices, and build enterprise-grade safety, audit, and human-in-the-loop controls. This is a senior remote individual-contributor position requiring deep production LLM experience, operational ownership, mentoring, and strong asynchronous written communication.

Role DNA

A quick view of the complexity, pace, ownership and collaboration implied by the job description.

Job Complexity

5/5
EasyHard

Pace & Pressure

5/5
RelaxedFast-paced

Autonomy Level

5/5
GuidedFull ownership

Communication Load

5/5
IndependentCollaborative
AI insightThis is a principal-level role covering complex distributed systems, production agentic AI, governance, and platform strategy across multiple teams and deployment models. The engineer must make consequential architectural decisions while remaining hands-on and participating in incident response.

Salary analysis

Estimated compensation compared with the broader US market for similar roles.

Estimated job median (per year)Market rate
$287,500
US market range (per year)$240k–$350k
AI insightThe yearly pay period is inferred from the job context; it was not explicitly stated by the employer. The employer discloses a US base salary range of $250,000–$325,000 USD per year; the midpoint is $287,500 annually. Estimated US market compensation for a principal engineer leading AI platform and distributed-systems architecture is approximately $240,000–$350,000 USD per year in base salary, excluding equity and bonuses.

Core skills

Skills and capabilities most closely associated with this opportunity.

Sample interview questions
How would you design an agent orchestration layer that remains reliable when tools fail, models time out, or requests are retried?

I would model execution as a durable session state machine with explicit idempotency keys, bounded retries, deadlines, cancellation propagation, and typed failure states. Tool calls would be isolated behind contracts with structured outputs and telemetry, while graceful degradation would return useful partial results or require human review rather than blocking the developer workflow.

What evaluation framework would you use to decide whether a change to an agent platform is safe to release?

I would combine versioned offline benchmark suites with controlled online experiments. The framework would measure task completion, correctness, tool-call accuracy, latency, cost per successful task, safety-policy adherence, and regression rates, with release gates for critical workflows and segmented analysis by model, customer configuration, and deployment type.

How would you support customer-defined agents and external tools without weakening platform security?

I would define a permissioned extension model using scoped identities, least-privilege tool grants, policy checks at invocation time, secrets isolation, audit trails, and explicit approval boundaries for sensitive actions. External integrations should use stable contracts and sandboxing where appropriate, with clear tenant-level controls over which agents, tools, and models are permitted.

Describe how you would reduce latency and cost for agentic workflows without materially reducing quality.

I would first instrument the end-to-end path to identify model, retrieval, tool, and orchestration bottlenecks. Likely approaches include routing requests to fit-for-purpose models, reducing redundant context, caching safe deterministic results, parallelizing independent tool calls, setting budget-aware stopping criteria, and validating quality through benchmark and online experiment results.

How do you create alignment on an ambiguous architecture decision across multiple teams when you have no direct authority?

I start by framing the customer problem, constraints, trade-offs, and measurable decision criteria in a concise design document. I involve affected teams early, make disagreement visible, use prototypes or data to resolve uncertain points, record the decision and rationale, and follow through on adoption with clear ownership and milestones.

This analysis is generated from the job description. Salary estimates, role characteristics and sample answers are guidance, not employer-provided facts.
Opportunity details

About this role.

GitLab is the intelligent orchestration platform for DevSecOps. GitLab enables organizations to increase developer productivity, improve operational efficiency, reduce security and compliance risk, and accelerate digital transformation. More than 50 million registered users and more than 50% of the Fortune 100* trust GitLab to ship better, more secure software faster.

The same principles built into our products are reflected in how our team works: we embrace AI as a core productivity multiplier, with all team members expected to incorporate AI into their daily workflows to drive efficiency, innovation, and impact. GitLab is where careers accelerate, innovation flourishes, and every voice is valued. Our high-performance culture is driven by our values and continuous knowledge exchange, enabling our team members to reach their full potential while collaborating with industry leaders to solve complex problems. Co-create the future with us as we build technology that transforms how the world develops software.

*Fortune 500® is a registered trademark of Fortune Media IP Limited, used under license. Claim based on GitLab data. Fortune 100 refers to the top 20% ranked companies in the 2025 Fortune 500 list, published in June 2025. Fortune and Fortune Media IP Limited are not affiliated with, and do not endorse products or services of GitLab.

About the role

We are looking for a Principal Engineer to help build GitLab Duo Agent Platform, the agentic orchestration layer that runs AI agents and flows across the entire software lifecycle.

Principal Engineers at GitLab are recognized experts across multiple technology domains and represent the senior technical leadership within their organization. In this role you will set technical direction for a significant part of the Agent Platform, working across several teams to turn agentic capabilities into something engineering organizations can actually depend on: predictable, observable, governable, and fast enough to sit in the critical path of real development work.

The Agent Platform is not a single service. It spans agent execution and orchestration, flow definition and event-driven triggering, the AI Catalog through which customers discover and manage agents, model routing and evaluation, and the governance surface that decides which agents may act where and under whose identity. It runs on GitLab.com, GitLab Dedicated, and Self-Managed installations, including deployments that use self-hosted models to meet compliance requirements. You will work on the parts of that system where the hard architectural questions live, and you will do it hands-on.

This role reports into the VP of the AI Engineering organization and works closely with Engineering Managers, Staff Engineers, Product Management, Infrastructure and Security.

What you’ll do

  • Define the technical direction for a significant area of Duo Agent Platform across several teams, and partner with Engineering Managers and Product to turn it into a roadmap with a six-month horizon.
  • Build the orchestration layer that makes agents and flows reliable: session lifecycle, tool invocation, context assembly, concurrency, retries, timeouts, and failure modes that degrade gracefully instead of stalling a developer’s workflow.
  • Hands on work on driving topics throughout the whole organisation from POC level, architecture to full implementation.
  • Participate in Incident Management on-call rotation, to help meet availability goals for GitLab.com.
  • Strong product builder and ownership mentality. Understanding of product strategy, technical strategy, delivering customer value and iteration towards a successful product
  • Write clear, opinionated design documents, architecture narratives, and decision records that let other teams make aligned decisions. Ensure the architecture holds across multi-tenant GitLab.com, Dedicated, and Self-Managed deployments, including self-hosted model configurations, and respects data governance boundaries in each.
  • Use the platform on our own work, find where it falls short of what we claim, and feed that back through improvements into the product.
  • Mentor Staff and Senior Engineers, and raise the bar for experimentation rigor and cross-team collaboration.
  • Work across the whole platform on different projects, features and teams. This reaches across:
    • Design the extensibility surface so customers can build custom agents and flows, connect their own tools and internal systems, and bring external agents such as Claude Code and Codex into GitLab workflows without weakening the platform’s guarantees.
    • Drive down the cost and latency of agentic requests, and work with Product on how model selection, routing, and credit consumption behave as usage grows.
    • Define the evaluation practice for the platform: offline benchmarks and online experiments that measure task completion, correctness, tool-call accuracy, latency, and cost per task, and that catch regressions before customers do.
    • Build the governance and safety surface with Security and Compliance: policy-driven controls over which agents and models may run where, permission and identity semantics that align to existing roles and group structure, human-in-the-loop checkpoints, traceability of every flow session, and audit that stands up to enterprise review.
  • Complete Interview Training and participate in hiring for the department as a technical interviewer or panel member.

What you’ll bring

  • 8+ years of software engineering experience, including 3+ years at Staff, Principal level or equivalent senior technical leadership.
  • Hands-on production experience building on large language models: agent loops, tool calling, context management, retrieval, prompt and model evaluation. Experience shipping this to real users, not only prototyping it.
  • Strong background in distributed systems, including service decomposition, fault tolerance, observability, and operational resilience under multi-tenant load.
  • A track record of defining evaluation frameworks and using benchmark and experiment results to make architecture decisions, rather than shipping on intuition.
  • Experience designing human-in-the-loop controls, safety guardrails, and responsible AI practices for systems that act on a customer’s behalf.
  • Ability to drive alignment across Engineering, Product, Infrastructure, and Security on problems that start out ambiguous.
  • Experience mentoring senior engineers and shaping technical direction across multiple teams without direct authority.
  • Comfort working in one or more of Ruby, Go, Python, or TypeScript, and willingness to work across the stack the platform spans.
  • Excellent written and asynchronous communication, and the ability to work effectively in a fully remote, globally distributed organization.

About the team

The AI engineering organisation is building the product and infrastructure that make GitLab an effective environment for running agents across the software development lifecycle. We’re distributed across the world and partner with teams throughout GitLab to help the GitLab Duo Agent Platform become the backbone of GitLab’s AI features. You will report directly to the VP of Engineering for AI and especially work together with all the teams, the engineering management and other principal engineers in the AI department. This is an opportunity to lead a consequential group as AI changes how software is built.

The base salary range for this role’s listed level is currently for residents of the United States only. This range is intended to reflect the role’s base salary rate in locations throughout the US. Grade level and salary ranges are determined through interviews and a review of education, experience, knowledge, skills, abilities of the applicant, equity with other team members, alignment with market data, and geographic location. The base salary range does not include any bonuses, equity, or benefits. See more information on our benefits and equity. Sales roles are also eligible for incentive pay targeted at up to 100% of the offered base salary.

United States Salary Range

$250,000—$325,000 USD

How GitLab Supports Full-Time Employees

Please note that we welcome interest from candidates with varying levels of experience; many successful candidates do not meet every single requirement. Additionally, studies have shown that people from underrepresented groups are less likely to apply to a job unless they meet every single qualification. If you’re excited about this role, please apply and allow our recruiters to assess your application.


Country Hiring Guidelines: GitLab hires new team members in countries around the world. All of our roles are remote, however some roles may carry specific location-based eligibility requirements. Our Talent Acquisition team can help answer any questions about location after starting the recruiting process.

Privacy Policy: Please review our Recruitment Privacy Policy. Your privacy is important to us.

GitLab is proud to be an equal opportunity workplace and is an affirmative action employer. GitLab’s policies and practices relating to recruitment, employment, career development and advancement, promotion, and retirement are based solely on merit, regardless of race, color, religion, ancestry, sex (including pregnancy, lactation, sexual orientation, gender identity, or gender expression), national origin, age, citizenship, marital status, mental or physical disability, genetic information (including family medical history), discharge status from the military, protected veteran status (which includes disabled veterans, recently separated veterans, active duty wartime or campaign badge veterans, and Armed Forces service medal veterans), or any other basis protected by law. GitLab will not tolerate discrimination or harassment based on any of these characteristics. See also GitLab’s EEO Policy and EEO is the Law. If you have a disability or special need that requires accommodation, please let us know during the recruiting process.

Apply now >

This job listing has been manually reviewed by the Jobicy Trust & Safety Team for compliance with our posting guidelines, including verification of the company's legitimacy, accuracy of job details, clarity of remote work policy, and absence of misleading or fraudulent content.

Next step

Apply now.

Follow the employer’s application method and review Jobicy’s safety guidance before sharing personal information.

Did you apply?Let us know, and we’ll help you track your application.

Continue on the employer website

Protect your personal information and never pay to secure an interview or job offer. .

Log in to save
One quick step before you apply

Sign in to continue.

Sign in or create a free account to continue to the employer's application.

Add alert
Jobs Talent AI Tools Salaries
Menu