[All remote jobs](https://jobicy.com/jobs.md)[![Smartsheet logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/8b6103bd-221.jpg)](https://jobicy.com/company/smartsheet.md)[Smartsheet](https://jobicy.com/company/smartsheet.md)

# Principal Software Engineer – Observability & Telemetry Data

AI Summary This principal-level software engineering role defines the architecture for Smartsheet's observability and telemetry data platform. It covers large-scale telemetry ingestion, OpenTelemetry standards, lakehouse data modeling, Databricks and MLflow integration, and data-driven reliability analytics. The engineer will influence multiple platform organizations, establish technical standards, mentor engineers, and lead complex production incident resolution. The position requires deep hands-on expertise across distributed systems, cloud infrastructure, streaming pipelines, data platforms, and observability economics.[Apply for this job](#job-application)[View company](https://jobicy.com/company/smartsheet.md)ShareRemote from[USA](https://jobicy.com/job-region/usa.md)SalaryUSD 222,500–257,500 / yrDepartment[Software Engineering](https://jobicy.com/categories/engineering.md)EmploymentFull TimeExperienceSeniorPublished16 Nov 2026Apply before16 Dec 2026Listing views26Application actions0Application toolkit

## Make your next move.

Prepare your resume, explore your fit, and draft a cover letter for this opportunity.

Opportunity details

## About this role.

For over 20 years, [Smartsheet](https://jobicy.com/company/smartsheet.md) has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking. That is magic at work, and it’s what we show up for every day.

Observability Engineering owns the breadth and depth of observability and telemetry at Smartsheet. We build and operate the shared platform every engineering team depends on to understand how their software behaves in production (metrics, logs, distributed traces, RUM, synthetics, and AI/agentic telemetry) across a global, multi-region, commercial and government footprint. We are consolidating ownership of all of it into a single correlated platform, and treating telemetry as what it actually is: one of the largest and most valuable datasets at the company. It’s a broad mandate with real scope to shape, and we hold this internal platform to the same rigor, reliability, and product standards as customer-facing software.

As a Principal Software Engineer (Observability & Telemetry Data), you will set the technical direction for how Smartsheet collects, models, stores, and derives value from telemetry data. This is an architect role: you will define the telemetry data platform and the standards that govern it, carry that architecture across engineering org boundaries, partner with our Data and AI Platform teams on Databricks and MLflow integration, and act as the company’s definitive technical voice on observability. You will influence roadmaps you do not own, resolve the hard design trade-offs between fidelity and cost, and raise the technical bar for every engineer instrumenting a service.

This full-time position reports to the Team Lead, Observability Engineering and can be located in our Bellevue, WA office, or you may work remotely from anywhere in the US where Smartsheet is a registered employer.

You Will

* Architect the Telemetry Data Platform: Own the end-to-end design for how telemetry is landed, modeled, and queried, including open table formats, partitioning and schema evolution strategy, separation of storage and compute, tiered retention, and the query interfaces engineers actually use. Make telemetry a durable, portable, Smartsheet-owned dataset rather than a vendor-locked byproduct.
* Own the Telemetry Data Model: Define the semantic conventions, shared business identifiers (user, org, plan, tenant), and schema standards that let any signal be correlated with any other, and drive their adoption across every service team at Smartsheet.
* Set OpenTelemetry Direction: Lead the migration to OTel-based instrumentation, defining collector architecture, context propagation, and sampling strategy (including tail-based sampling) so that engineers can move from a log line to a trace to a metric without losing the thread.
* Integrate AI and Agentic Telemetry with Databricks: Own the architecture connecting our observability platform to Databricks and MLflow, so that agentic and model telemetry (prompt, completion, tool and MCP calls, evaluation results) is captured un-sampled, stays useful under the input/output redaction our governance requires, and reconciles cleanly with the traces and metrics in our primary observability stack.
* Instrument the Data Platform Itself: Bring first-class observability to our data estate, including Databricks jobs, pipelines, and warehouses, with meaningful signals for freshness, data quality, lineage, and cost, so data reliability is measured with the same discipline as service reliability.
* Build the Analytics Layer on Telemetry: Turn telemetry into decision-grade analytics, covering reliability and incident metrics, telemetry cost and chargeback models, and adoption and coverage reporting that leadership can act on.
* Engineer Collection and Routing at Scale: Architect the high-volume collection and routing tier (FluentBit, Kinesis, and OTel collectors) that moves telemetry from every service to its destination across US, EU, AU, and GovCloud regions, and own the migration of these pipelines as we consolidate onto a unified backend.
* Own Telemetry Economics: Set the cost architecture for observability data, including ingest governance, cardinality control, and storage tiering, so that teams get the fidelity they need to debug without the spend pressure that causes them to under-instrument.
* Carry Architecture Across Org Boundaries: Partner with the Data Platform, AI Platform, and infrastructure organizations to align telemetry architecture with theirs, influence roadmaps you do not own, and represent observability in company-level platform and vendor decisions.
* Raise the Technical Bar: Lead design and code reviews, author the architecture decisions and standards others build against, and mentor senior and mid-level engineers on instrumentation, telemetry data modeling, and cost-aware design.
* Participate in a production support and on-call rotation, taking ownership of the most complex issue resolution and driving root-cause analysis that improves system resiliency.

You Have

* 10+ years of experience building and operating large-scale distributed systems, data platforms, or observability infrastructure, including time at Principal or Staff level.
* Deep Observability Expertise: Hands-on production ownership of metrics, logs, and distributed tracing at scale, including at least one major backend (Datadog, or comparable) and a working understanding of cardinality and cost mechanics.
* Telemetry Data Engineering: Demonstrated depth in large-scale data architecture, including open table formats (Delta Lake, Apache Iceberg), Spark or comparable distributed processing, streaming ingestion, partitioning and schema evolution, and query performance and cost tuning over very large datasets.
* Databricks Depth: Practical experience with the Databricks platform (jobs, clusters, Unity Catalog, Delta) and with MLflow for model and agent telemetry.
* OpenTelemetry Depth: Practical experience with OTel collectors, semantic conventions, context propagation, and sampling strategy, including tail-based sampling.
* Pipeline Engineering: Experience with high-volume log and telemetry pipelines, including FluentBit or Fluentd, streaming transport such as Kinesis or Kafka, and search backend index and mapping design.
* Advanced AWS & Kubernetes Expertise: EKS, ECS Fargate, EC2, Lambda, and CloudWatch in production.
* 10+ years of programming experience with modern languages such as Go, Java, Python, or Scala, and strong SQL.
* Infrastructure as Code: Terraform, and GitOps workflows such as Flux or ArgoCD.
* Architectural Influence at Scale: A track record of setting technical direction that multiple teams and organizations adopted, including the written artifacts (architecture decisions, standards, RFCs) that made it durable after you moved on.
* Strong incident response instincts, with experience improving mean-time-to-resolution through better instrumentation and better data rather than more heroics.
* A degree in Computer Science, Engineering, or a related field, or equivalent practical industry experience.
* Legally eligible to work in the U.S. on an ongoing basis

Nice to Have

* Experience instrumenting LLM or agentic systems, including OTel GenAI semantic conventions and tracing agent tool-call workflows.
* Data reliability engineering practice: freshness, quality, and lineage SLOs for production data pipelines
* Telemetry cost engineering or FinOps at scale.
* Experience in regulated environments (FedRAMP, GovCloud) and designing telemetry that remains useful under redaction.
* Prometheus, Grafana, and Alertmanager.
* Snowflake, or experience operating across more than one lakehouse or warehouse platform.
* Service catalog or internal developer platform work (Backstage or similar).

Current US Perks & Benefits:

* Employer subsidized medical/vision and dental coverage for full-time employees
* 401k Match to help you save for your future (50% of your contribution up to the first 6% of your eligible pay)
* Monthly stipend to support your work and productivity
* Flexible Time Away Program, plus Sick Time Off
* US employees are automatically covered under Smartsheet-sponsored life insurance, short-term, and long-term disability plans
* US employees receive 12 paid holidays per year
* Up to 24 weeks of Parental Leave
* Personal paid Volunteer Day to support our community
* Opportunities for professional growth and development including access to Udemy online courses
* Company Funded Perks, including a counseling membership, local retail discounts, and your own personal Smartsheet account
* Teleworking options from any registered location in the U.S. (role specific)

Smartsheet provides a competitive base salary range for roles that may be hired in different geographic areas we are licensed to operate our business from. Actual compensation is determined by several factors including, but not limited to, level of professional, educational experience, skills, and specific candidate location. In addition, this role will be eligible for a market competitive incentive opportunity.

US Base Salary Pay Range

$222,500—$257,500 USD

Get to Know Us:

At Smartsheet, your ideas are heard, your potential is supported, and your contributions have real impact. You’ll have the freedom to explore, push boundaries, and grow beyond your role. We welcome diverse perspectives and nontraditional paths—because we know that impact comes from individuals who care deeply and challenge thoughtfully. When you’re doing work that stretches you, excites you, and connects you to something bigger, that’s magic at work. Let’s build what’s next, together.

Equal Opportunity Employer:

Smartsheet is an Equal Opportunity (EEO) employer committed to fostering an inclusive environment with the best employees. It is our policy to provide equal employment opportunities to all qualified applicants in accordance with applicable laws in the US, UK, Australia, Germany, Costa Rica, Japan, Bulgaria, India, and Singapore. All qualified applicants will receive consideration without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran or disabled status, or genetic information.

If there are preparations we can make to help ensure you have a comfortable and positive interview experience, please let us know.

#LI-Remote

Show more

[Apply now >](https://jobicy.com/jobs/162553-principal-software-engineer-observability-telemetry-data.md)

*

![Upload CV](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI2NSIgaGVpZ2h0PSI2NSIgZmlsbD0ibm9uZSIgeG1sbnM6dj0iaHR0cHM6Ly92ZWN0YS5pby9uYW5vIj48ZyBjbGlwLXBhdGg9InVybCgjQSkiPjxwYXRoIGQ9Ik0wIDBINjVWNjVIMFYwWiIgZmlsbD0iIzAyOWFlYiIvPjxnIGZpbGw9IiNmZmYiIHN0cm9rZT0iI2ZmZiIgc3Ryb2tlLXdpZHRoPSIyIj48cGF0aCBkPSJNMzMuMDQ5IDE1LjQ1NGExLjQzIDEuNDMgMCAwIDAtMi4wOTcgMGwtNy41NzkgOC4xNDdhMS4zOCAxLjM4IDAgMCAwIC4wOSAxLjk3MyAxLjQ0IDEuNDQgMCAwIDAgMi4wMDgtLjA4OGw1LjEwOS01LjQ5MnYyMC42MWExLjQxIDEuNDEgMCAwIDAgMS40MjEgMS4zOTdjLjc4NSAwIDEuNDIxLS42MjUgMS40MjEtMS4zOTd2LTIwLjYxbDUuMTA5IDUuNDkyYTEuNDQgMS40NCAwIDAgMCAyLjAwOC4wODggMS4zOCAxLjM4IDAgMCAwIC4wOS0xLjk3M2wtNy41NzktOC4xNDZ6TTE2Ljc2OSAzOC40YzAtLjc3My0uNjItMS40LTEuMzg1LTEuNFMxNCAzNy42MjcgMTQgMzguNHYuMTAybC4yMTUgNi4yMjljLjIyMyAxLjY4LjcwMSAzLjA5NSAxLjgxMyA0LjIxOHMyLjUxIDEuNjA3IDQuMTcyIDEuODMzYzEuNi4yMTggMy42MzYuMjE4IDYuMTYuMjE4aDExLjI4bDYuMTYtLjIxOGMxLjY2Mi0uMjI2IDMuMDYxLS43MDkgNC4xNzItMS44MzNzMS41ODktMi41MzggMS44MTMtNC4yMThDNTAgNDMuMTEzIDUwIDQxLjA1NSA1MCAzOC41MDNWMzguNGMwLS43NzMtLjYyLTEuNC0xLjM4NS0xLjRzLTEuMzg1LjYyNy0xLjM4NSAxLjRsLS4xOSA1Ljk1OGMtLjE4MiAxLjM3LS41MTUgMi4wOTUtMS4wMjYgMi42MTJzLTEuMjI4Ljg1My0yLjU4MyAxLjAzOGMtMS4zOTUuMTktMy4yNDMuMTkzLTUuODkzLjE5M0gyNi40NjJjLTIuNjUgMC00LjQ5OC0uMDAzLTUuODkzLS4xOTMtMS4zNTUtLjE4NC0yLjA3Mi0uNTIxLTIuNTgzLTEuMDM4cy0uODQ0LTEuMjQyLTEuMDI2LTIuNjEyYy0uMTg3LTEuNDEtLjE5MS0zLjI3OS0uMTkxLTUuOTU4eiIvPjwvZz48L2c+PGRlZnM+PGNsaXBQYXRoIGlkPSJBIj48cGF0aCBmaWxsPSIjZmZmIiBkPSJNMCAwaDY1djY1SDB6Ii8+PC9jbGlwUGF0aD48L2RlZnM+PC9zdmc+)

### Upload your resume now

To unlock remote work opportunities and be discovered by global employers.

This job listing has been manually reviewed by the Jobicy Trust & Safety Team for compliance with our posting guidelines, including verification of the company's legitimacy, accuracy of job details, clarity of remote work policy, and absence of misleading or fraudulent content.

## Role DNA

A quick view of the complexity, pace, ownership and collaboration implied by the job description.

### Job Complexity

5/5EasyHard

### Pace & Pressure

5/5RelaxedFast-paced

### Autonomy Level

5/5GuidedFull ownership

### Communication Load

5/5IndependentCollaborative

AI insightThe role combines principal-level architectural ownership with broad cross-organizational influence over high-volume, multi-region observability systems. It requires rare depth across telemetry, distributed data engineering, cloud platforms, AI telemetry, and cost-aware platform design.

## Salary analysis

Estimated compensation compared with the broader US market for similar roles.

Estimated job median (per year)Market rate$240,000US market range (per year)$215k–$280k0$308k

AI insightThe yearly pay period is inferred from the job context; it was not explicitly stated by the employer. The employer discloses a USD 222,500–257,500 base salary range. The annual period is inferred because the posting does not explicitly state a cadence, but the amount and full-time US principal-engineering context strongly indicate annual base pay. The estimated US market range for a principal observability and data-platform engineer is USD 215,000–280,000 per year, excluding incentive compensation and equity.

## Core skills

Skills and capabilities most closely associated with this opportunity.

[Distributed systems architecture](https://jobicy.com/jobs?search_keywords=Distributed%20systems%20architecture.md)[Observability engineering](https://jobicy.com/jobs?search_keywords=Observability%20engineering.md)[OpenTelemetry](https://jobicy.com/jobs?search_keywords=OpenTelemetry.md)[Telemetry data modeling](https://jobicy.com/jobs?search_keywords=Telemetry%20data%20modeling.md)[Databricks](https://jobicy.com/jobs?search_keywords=Databricks.md)[MLflow](https://jobicy.com/jobs?search_keywords=MLflow.md)[Delta Lake](https://jobicy.com/jobs?search_keywords=Delta%20Lake.md)[Apache Iceberg](https://jobicy.com/jobs?search_keywords=Apache%20Iceberg.md)[Apache Spark](https://jobicy.com/jobs?search_keywords=Apache%20Spark.md)[Streaming data pipelines](https://jobicy.com/jobs?search_keywords=Streaming%20data%20pipelines.md)[AWS](https://jobicy.com/jobs?search_keywords=AWS.md)[Kubernetes](https://jobicy.com/jobs?search_keywords=Kubernetes.md)[Terraform](https://jobicy.com/jobs?search_keywords=Terraform.md)[SQL](https://jobicy.com/jobs?search_keywords=SQL.md)[Incident response](https://jobicy.com/jobs?search_keywords=Incident%20response.md)Next step

## Apply now.

Follow the employer’s application method and review Jobicy’s safety guidance before sharing personal information.

Keep exploring

## Related remote jobs.

*
![Cologix, Inc. logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/ab469e6c-221-1.png)
Cologix, Inc. Nov 16  New

### [NetSuite Developer](https://jobicy.com/jobs/162552-netsuite-developer.md)

About Our Company: Headquartered in Denver, Colorado, Cologix is a leading North America network-neutral interconnection and hyperscale edge data center company. Our platform gives customers access to 45+ digital edge…

*
![Oklo Inc. logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2026/11/577cd1a72e58-221.webp)
Oklo Inc. Nov 16  New

### [Safety Analysis Engineer](https://jobicy.com/jobs/162537-safety-analysis-engineer.md)

Thanks for your interest in Oklo! We are searching for a Safety Analysis Engineer to join our team. Position Description:We are seeking a skilled Safety Analysis Engineer to support the…

*
![Rush Street Interactive logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2026/11/7c2df01086a1-221.webp)
Rush Street Interactive Nov 16  New

### [Senior Software Engineer, Services](https://jobicy.com/jobs/162522-senior-software-engineer-services.md)

Rush Street Interactive (NYSE: RSI) is a market leader in online casino and sports betting, currently operating real-money gaming with our brands: BetRivers.com, PlaySugarHouse.com, and RushBet.co. We’re building bridges between…

*
![GitLab logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2020/12/WRILS-201207055737-109952.jpg)
GitLab Nov 16  New

### [Backend Engineer (Python),AI Engineering: Agent Foundations](https://jobicy.com/jobs/155430-backend-engineer-pythonai-engineering-agent-foundations.md)

GitLab is the intelligent orchestration platform for DevSecOps. GitLab enables organizations to increase developer productivity, improve operational efficiency, reduce security and compliance risk, and accelerate digital transformation. More than 50…

*
![Playson logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/6f266d9c-221-1.png)
Playson Nov 16  New

### [Senior Backend Engineer (Boosters)](https://jobicy.com/jobs/153227-senior-backend-engineer-boosters.md)

About the Role Our promotional features team is seeking a Senior Backend Engineer. This role is focused on developing and optimising promotional features for our gaming platforms, where performance, scalability,…

*
![Palmstreet logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/fa38ec7c-221.png)
Palmstreet Nov 16  New

### [Senior+ Backend Software Engineer – Asia](https://jobicy.com/jobs/153195-senior-backend-software-engineer-asia.md)

About Palmstreet: Palmstreet is the #1 live shopping platform for rare plants, handmade crafts, & unique goods. Founded in 2020 by Instagram and Apple veterans, Palmstreet began as a plant…

*
![Playson logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/6f266d9c-221-1.png)
Playson Nov 16  New

### [Senior Software Engineer (Game Engine)](https://jobicy.com/jobs/146058-senior-software-engineer-game-engine.md)

About the RoleWe’re looking for a Senior Software Engineer to join a newly forming team that will own one of the most specialised and business-critical domains in our technology landscape…

*
![Canonical logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/b4d068a9-221-1.png)
Canonical Nov 16  New

### [Ubuntu Linux Kernel Test Engineer](https://jobicy.com/jobs/145995-ubuntu-linux-kernel-test-engineer.md)

Ubuntu is the most widely used Linux distribution in the world, delivering kernels across a vast matrix of versions, architectures, and configurations – with up to 15 years of security…

*
![Reka logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/4c8a0366-221.png)
Reka Nov 16  New

### [Member of Technical Staff (Robotics)](https://jobicy.com/jobs/153241-member-of-technical-staff-robotics.md)

We are an AI research lab building foundational intelligence for the physical world. Our models are designed to solve complex real-world challenges, powering everything from intelligent robots to next-generation wearables…

*
![Palmstreet logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/fa38ec7c-221.png)
Palmstreet Nov 16  New

### [Senior+ Backend Software Engineer – Canada](https://jobicy.com/jobs/153205-senior-backend-software-engineer-canada.md)

About Palmstreet: Palmstreet is the #1 live shopping platform for rare plants, handmade crafts, & unique goods. Founded in 2020 by Instagram and Apple veterans, Palmstreet began as a plant…

[Browse all jobs](https://jobicy.com/jobs.md)