About this role.
Gremlin is seeking a Senior Backend Engineer to build Chaos Engineering and reliability tooling for high-availability software teams. The role requires deep Java expertise, Go and systems-level programming exposure, and experience with cloud platforms, distributed systems, databases, and container infrastructure. The engineer will collaborate remotely with product and business stakeholders, deliver production features, mentor teammates, and advocate for customer-focused engineering quality. This is a US-based remote role supporting Gremlin's reliability platform and enterprise customers.
Role DNA
A quick view of the complexity, pace, ownership and collaboration implied by the job description.
Job Complexity
5/5Pace & Pressure
4/5Autonomy Level
4/5Communication Load
4/5Salary analysis
Estimated compensation compared with the broader US market for similar roles.
Core skills
Skills and capabilities most closely associated with this opportunity.
Sample interview questions
I would begin with strict authorization, scoped blast-radius controls, auditable experiment definitions, and default safety guardrails. The execution plane should be decoupled from the control plane, use idempotent operations and durable state, and include clear rollback or stop mechanisms. I would also prioritize observability, tenant isolation, and failure-mode testing before broad rollout.
I would explain the service architecture, critical dependencies, and the reliability risks identified through incidents or telemetry. I would describe concrete improvements such as timeouts, retries with backoff, circuit breakers, idempotency, better database access patterns, and automated tests. I would conclude with measurable results such as reduced error rates, improved latency, or fewer on-call escalations.
I assess invocation patterns, execution duration, concurrency behavior, cold-start sensitivity, networking requirements, operational ownership, and cost. Lambda is often effective for event-driven, bounded, independently scalable work, while containers better fit persistent processes, complex runtime dependencies, or workloads needing sustained connections and predictable performance. I validate the choice with load testing and operational requirements rather than using either model by default.
I would model access patterns intentionally, use conditional writes and optimistic concurrency where appropriate, and make consumers idempotent. For external side effects, I would use durable eventing or an outbox-style pattern, retries with bounded backoff, and dead-letter handling. Monitoring for lag, duplicate processing, failed writes, and reconciliation outcomes is essential.
I make mentorship part of normal delivery through thoughtful code reviews, design discussions, pairing, and clear documentation. I provide context about tradeoffs rather than only prescribing solutions, then progressively delegate ownership as confidence grows. This helps engineers develop judgment while keeping work aligned with team priorities and delivery commitments.
Today’s complex, fast-paced systems have become a minefield of reliability risks—any of which could cause an outage that costs millions and destroys customer confidence. That’s why high-availability teams use the Gremlin to find and fix reliability risks before they become incidents.
Gremlin Reliability Platform helps software teams proactively monitor and test their systems for common reliability risks, build and enforce reliability standards, and automate their reliability practices organization-wide. As the industry leader in Chaos Engineering and reliability testing, we work with hundreds of the world’s largest organizations where high availability is non-negotiable.
About the Role of the Senior Software Engineer
As a Software Engineer at Gremlin, you will have the opportunity to improve the reliability of the internet at large by developing Chaos Engineering tooling. You will be able to leverage your engineering experience to inform product design as well as solve complex technical problems that directly impact our customers (which range from the Fortune 500 to smaller organizations). You will work closely with a small, talented team focused on quality, delivery, and predictability.
In this role, you’ll get to:
- Work closely with engineers, product managers, and other stakeholders to design and build the latest and greatest in Chaos Engineering tooling
- Leverage strong collaboration and communication skills to deliver new features within a remote culture
- Partner with product and other business units to understand business problems and present technical solutions and tradeoffs
- Actively mentor and grow your teammates
- Care deeply about the customer experience
We’ll expect you to have:
- 5+ years professional Java software engineering experience
- Experience in Go & Systems Level Programming
- Experience in cloud technologies: e.g AWS, Lambda, Serverless. Experience with other cloud technologies like Google, Oracle also considered
- Experience in DynamoDB and/or other no-sql DB or experience in any major relational databases
- Experience in infrastructure & systems level technologies: e.g., Linux, Docker, Kubernetes, OpenShiftExperience in architecting complex distributed systems and integrating with external systems
- Strong advocate and practitioner of automated testing, CI/CD, and engineering best practices
Bonus Experience:
- Has been on-call and participated in an incident management program
- Familiarity with modern JavaScript frameworks & web development practices: e.g., React, TypeScript, etc.
- Experience taking features from concept to full production release
*The role does not offer sponsorship employment benefits.
**If you don’t think you meet all of the criteria below but still are interested in the job, please apply. Nobody checks every box—we’re looking for candidates that are particularly strong in a few areas, and have some interest and capabilities in others.
Compensation
We expect the salary range for this role to be $220,000 – $290,000. We recognize that salary varies from person to person depending on level of experience and we welcome direct conversations about it. The final offer will vary based on assessment of a candidate’s skills and ability and our budget and market data.
Gremlin offers competitive total compensation packages including 401k Matching, Equity and other benefits such as flexible time off and paid company holidays.
About Gremlin:
Gremlin is a team of industry veterans and people eager to learn from one another. We set the standard for reliability and equip leading organizations with the mindset and expertise needed to drive reliability improvements that move the world forward. We’re backed by top-tier investors Index Ventures, Amplify Partners, and Redpoint Ventures. Our customers love us, and we’re thrilled to be a partner in their success.
What Do We Care About:
- We Care about our People
People are our critical differentiators. The company strives to treat our people with respect, empathy, and dignity. We expect that our people will treat each other similarly. In both cases, we will assume good intent. All are welcome at Gremlin. We know our differences make us stronger and that our best ideas and contributions can come from anyone at any level. - We Care about Collaboration
Gremlin is strongest when we come together as one team with shared goals. Be the glue, not the glitter. But as a remote company, teamwork and collaboration won’t happen by accident. We approach every challenge as a shared challenge. We rely on each other for diverse perspectives and creative ideas. We celebrate our wins as a team. - We Care about Results
Be high productivity, low drama. Results matter. To keep our pace, everyone owns the outcomes of their actions and takes action when needed. We reward speed over perfection. We empower each other to iterate and experiment.You are welcome at Gremlin for who you are. The more voices and ideas we have represented in our business, the more we will all flourish, contribute, and build a more reliable internet. Gremlin is a place where everyone can grow and is encouraged. However you identify and whatever background you bring with you, please apply if this sounds like a role that would make you excited to come into work everyday. It’s in our differences that we will find the power to keep building a more reliable internet by building and designing tools used by the best companies in the world.
You are welcome at Gremlin for who you are. The more voices and ideas we have represented in our business, the more we will all flourish, contribute, and build a more reliable internet. Gremlin is a place where everyone can grow and is encouraged. However you identify and whatever background you bring with you, please apply if this sounds like a role that would make you excited to come into work everyday. It’s in our differences that we will find the power to keep building a more reliable internet by building and designing tools used by the best companies in the world.
Visit our website to learn more – https://www.gremlin.com/about
Upload your resume now
- To unlock remote work opportunities and be discovered by global employers.
This job listing has been manually reviewed by the Jobicy Trust & Safety Team for compliance with our posting guidelines, including verification of the company's legitimacy, accuracy of job details, clarity of remote work policy, and absence of misleading or fraudulent content.
