# Principal Engineer – AI Platform Remote from[USA](https://jobicy.com/job-region/usa.md)Salary, yearly, USD 204,000 - 260,000Department [Software Engineering](https://jobicy.com/categories/engineering.md) Employment type Full Time, Job posted21 Jun 2026Apply before21 Jul 2026Experience level Senior Views / Applies 1667 / 102 [About company](https://jobicy.com/company/iherb.md) * Share About [iHerb](https://jobicy.com/company/iherb.md) On a mission to make health and wellness accessible to all. [Health, Wellness & Fitness](https://jobicy.com/company-category/health-wellness-fitness.md) * 1996 Actively Hiring Verified job posting This job post has been [manually reviewed](https://jobicy.com/tools/help-center/employee/how-does-jobicy-verify-the-legitimacy-of-remote-job-listings.md) for authenticity and compliance. ### AI Summary iHerb is seeking a Principal Engineer for its AI Platform, responsible for defining the technical strategy for AI-native features such as a Wellness Agent, LLM-driven recommendations, and generative content. This role involves building shared AI infrastructure including RAG pipelines, model lifecycle management, evals frameworks, and guardrails. The ideal candidate will act as a hands-on contributor while also setting technical standards, mentoring senior engineers, and driving cross-org initiatives. This position requires expertise in large-scale distributed systems and a track record of using AI-assisted development tools like Claude Code or GitHub Copilot. ### Role DNA Job Complexity Easy Hard Pace & Pressure Relaxed Fast-paced Autonomy Level Guided Full Ownership Communication Load Independent Highly Collaborative AI Insight This role demands a rare combination of deep technical expertise in AI infrastructure, hands-on coding, and broad organizational influence across multiple teams and seniority levels. ### Salary Analysis Median Market Rate USD232,000US Market USD180k – 320k 0 USD352k AI Insight The offered salary range of $204,000 to $260,000 is competitive for a Principal Engineer in AI platform, aligning with the top percentile of the market. The median of $232,000 reflects the seniority and impact of the role. ### Core Skills Required [AI Platform Architecture](https://jobicy.com/jobs?search_keywords=AI+Platform+Architecture.md) [Distributed Systems](https://jobicy.com/jobs?search_keywords=Distributed+Systems.md) [Machine Learning](https://jobicy.com/jobs?search_keywords=Machine+Learning.md) [RAG Pipelines](https://jobicy.com/jobs?search_keywords=RAG+Pipelines.md) [MLOps](https://jobicy.com/jobs?search_keywords=MLOps.md) [Leadership](https://jobicy.com/jobs?search_keywords=Leadership.md) [Cross-team Collaboration](https://jobicy.com/jobs?search_keywords=Crossteam+Collaboration.md) [API Design](https://jobicy.com/jobs?search_keywords=API+Design.md) [Vector Search](https://jobicy.com/jobs?search_keywords=Vector+Search.md) [GenAI](https://jobicy.com/jobs?search_keywords=GenAI.md) ### Cover Letter Sample I am writing to express my strong interest in the Principal Engineer - AI Platform position at iHerb. With over a decade of experience in distributed systems and AI infrastructure, I have a proven track record of designing scalable platforms and leading cross-functional technical initiatives. My expertise in building RAG pipelines, MLOps, and guardrails aligns perfectly with the requirements of this role. I thrive at the intersection of technical execution and organizational influence, and I am excited about the opportunity to drive iHerb's AI platform strategy. I have successfully mentored senior engineers and established architectural standards that compound over time. I look forward to contributing to your team and helping iHerb deliver production-grade AI experiences. Copy ### Sample Interview Questions Describe your experience designing and deploying large-scale AI platform systems. What architectural decisions were most critical?I led the design of a multi-modal RAG platform serving millions of queries daily. Key decisions included choosing a hybrid vector-keyword search architecture to balance relevance and latency, and implementing a hierarchical embedding pipeline to reduce index rebuild times. I also set up a robust eval framework with A/B testing to ensure model quality in production.How would you approach building a shared AI platform layer that serves multiple product features and internal teams?I would start by defining clear interfaces and contracts for the platform, such as standardized retrieval APIs and model-serving pipelines. I'd prioritize observability and guardrails, and collaborate with product teams to understand their use cases. An iterative approach with early proof-of-concepts helps validate assumptions before scaling.Explain a time you drove a cross-cutting technical decision that affected multiple teams. How did you build consensus?At my previous company, I championed a unified ML feature store to avoid duplication across teams. I organized architecture reviews, presented trade-offs in terms of cost and development velocity, and ran a pilot with two teams. By demonstrating quantifiable benefits and incorporating feedback, I gained buy-in from all stakeholders.What is your philosophy on using AI-assisted development tools in a production environment?I believe AI assistants like Claude Code can significantly boost productivity if used with proper guardrails. I advocate for a 'human-in-the-loop' approach where AI generates initial code or suggestions, but engineers always review for correctness and security. We should also maintain a shared set of skills and automation patterns to ensure consistency.How do you ensure that the AI platform's architecture remains maintainable and scalable over time?I enforce strong modularity and clear separation of concerns, coupled with thorough documentation and architecture decision records. Regular architecture reviews and pair programming sessions help propagate knowledge. I also advocate for automated testing at all levels and invest in observability to catch regressions early. Job Summary: The Principal Engineer operates at the intersection of deep technical execution and broad organizational influence. This role will resolve the hardest architectural questions, set the AI-native engineering bar, and ensure that iHerb’s platform decisions compound over time rather than accumulate debt. Reporting to the VP of Engineering this role collaborates closely with peer Principals, Engineering Managers, and peer VPs across the organization. This role owns the end-to-end technical strategy for iHerb’s AI Platform and GenAI product experiences. This spans the user-facing AI layer (Wellness Agent, LLM-driven recommendations, RAG over catalog and reviews, generative content) and the shared AI infrastructure (RAG pipelines, evals framework, MLOps, guardrails, observability) that makes AI features production-grade and repeatable across the engineering organization. Job Expectations: * Define and own the AI platform architecture: retrieval infrastructure, model lifecycle, evals framework, guardrails, and GenAI product feature design. * Lead the build of the shared AI platform layer consumed by all AI product features and reusable by internal business teams. * Hands-on contributor: build production AI systems, write proofs of concept, and validate architecture through working software. * Set and enforce technical standards for the AI Platform team; drive architecture reviews and model quality reviews. * Coordinate with the Personalization team to define clear boundaries between the GenAI product layer and existing ML personalization infrastructure. * Contribute AI platform-specific patterns and lessons into iHerb’s shared AI-driven SDLC golden path. * Drive the hardest cross-cutting technical decisions across multiple teams and shared platform services. * Establish and evolve iHerb’s AI-driven SDLC golden path: shared standards, Claude Code skills, guardrails, and automation patterns. * Lead complex multi-team technical efforts by coordinating architecture reviews, aligning peer Principals and EMs, and resolving competing approaches. * Mentor and raise the technical bar across the engineering organization through code review, architecture review, and direct coaching of senior engineers. * Represent engineering in cross-functional conversations with product, data science, security, and infrastructure. * Feed architectural decisions into the shared knowledge base so institutional knowledge compounds across the organization. The duties and responsibilities described above may provide only a partial description of this position. This is not an exhaustive list of all aspects of the job. Other duties and responsibilities not outlined in this document may be added as necessary or desirable, with or without notice. Knowledge, Skills and Abilities: Required: * AI-driven SDLC (required): demonstrated use of AI-assisted development tools such as Claude Code, GitHub Copilot, or Cursor to ship production systems. Can articulate workflow changes, quality tradeoffs, and guardrail strategies. * Architecture at scale: experience designing and evolving large-scale distributed systems across multiple teams and years: APIs, data pipelines, event-driven architectures, or high-traffic platforms. * Cross-org technical leadership: track record of driving architectural standards, technical roadmaps, or platform initiatives that span multiple teams or organizations. * Engineering quality mindset: deeply held opinions on code quality, observability, CI/CD, test automation, and maintaining velocity without accumulating hidden debt. * Communication and influence: able to write clear architecture documents, present to technical and non-technical audiences, and build consensus without formal authority. * Experience working in distributed teams across the US, China, and Latin America. * Production experience designing and operating AI platform systems: RAG pipelines, vector search, embedding infrastructure, or retrieval-augmented applications at scale. * Hands-on experience with LLM evaluation: building eval frameworks, defining quality metrics, and iterating on model and prompt quality using data. * MLOps experience: model deployment, monitoring, lifecycle management, and cost governance in a production environment. * Experience building and operating agentic systems using frameworks such as LangChain, LlamaIndex, or equivalent. * GenAI in production is required; prototype or research-only experience is not sufficient at this level. * High degree of accuracy and attention to detail * Excellent organization skills and ability to multi-task Equipment Knowledge: * Experience with Microsoft Office Suite (Word, Excel, PowerPoint) * Experience with Google Business Suite (Gmail, Drive, Docs, Sheets, Forms) preferred Experience Requirements: Generally requires a minimum of 10+ years of software engineering experience, with a significant portion at senior, staff, or principal IC level. Education Requirements: Bachelor’s Degree in Computer Science or related field preferred, or a combination of education and equivalent work experience. Judgment/Reasoning Ability: Able to identify, troubleshoot and resolve problems quickly using sound judgment, poise and diplomacy. Ability to use judgment and reasoning skills, and determine when to escalate issues, as required, in a timely manner. Physical Demands: The physical demands described here are representative of those that must be met by a Team Member to successfully perform the essential functions of this job. While performing the duties of this job, the Team Member is regularly required to talk and hear. The Team Member is frequently required to sit, walk, climb stairs, use hands and fingers, bend, stoop and reach with hands and arms. Reaching above shoulder heights, below the waist or lifting as required to file documents or store materials throughout the work day. The Team Member may occasionally lift or move office products and supplies up to 25 pounds. Proper lifting techniques required. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. Work Environment: The noise in the work environment is usually moderate. Other factors are: * Hectic, fast-paced with multi-level distractions * Professional, yet casual work environment * Office / Warehouse environment * Ability to work extended hours as required #LI-JC1 The anticipated pay scale for this position can be found below, however the pay range applicable to you may vary by geographic location based on where the job is located or where you work. The final pay offered to a successful candidate will be dependent on several factors that may include but are not limited to the type and years of experience within the job, the type of years and experience within the industry, education, etc. iHerb, LLC is a multi-state employer and this pay scale may not reflect positions that work in other states or locations. Employees (and their families) that meet eligibility criteria as outlined in applicable plan documents are eligible to participate in our medical, dental, vision, and basic life insurance programs and may enroll in our company’s 401(k) plan. Employees will also be eligible for Time Off and Paid Sick Leave pursuant to the company’s policies. Employees will enjoy paid holidays throughout the calendar year. Eligibility requirements for these benefits will be controlled by applicable plan documents.Hired applicant may be awarded Restrict Stock Units and receive annual bonuses pursuant to eligibility and performance criteria defined in the respective plan documents and policies. For more information on iHerb benefits, visit us at [iHerbBenefits.com](https://www.iherbbenefits.com/). Anticipated Pay Scale:$204,000—$260,000 USD Staffing Agency Submission Notice iHerb does not accept unsolicited 3rd party (“Agency”) candidates. If you are an Agency, please send any requests to be considered as a supplier in our Vendor Management System to [staffingvendors@iherb.com](mailto:staffingvendors@iherb.com). Do not contact iHerb employees directly. If requested to work on a role, any Agency candidates would be presented through the internal recruiting organization. About iHerb iHerb is on a mission to make health and wellness accessible to all. We offer Earth’s best-curated selection of health and wellness products, at the best possible value, delivered with the most convenient experience. We’re the world’s largest eCommerce platform dedicated to vitamins, minerals, and supplements, and other health and wellness products. For more than 25 years, we’ve been making it simple for people all over the world to purchase the highest quality products. From supplements to skincare to grocery items, we ship over 50,000 products, from over 1,800 brands direct to our customers in 180+ countries. Our vision is to become the #1 destination for health and wellness across the world. With a passion for wellness and a mind for innovative solutions, iHerb team members share a vision for a healthier world that drives them each day. Our 5 Shared Values unite our global team: Focus on the Customer · Empower Our People · Be Entrepreneurial & Pivot Quickly · Embrace Diversity & Inclusion · Strive for Simplicity iHerb Benefits At iHerb, we are dedicated to offering programs designed to help our employees and their families stay healthy, live well, and plan for their financial future. Built on a strong foundation, our programs provide options and upgrades with flexibility, protection, and security in mind. For the comprehensive benefits list, visit [www.iHerbBenefits.com](http://www.iherbbenefits.com/). For our international team members, you may be eligible for benefits depending on the country where you are employed. The Talent Acquisition Partner/local HR representative will go over the benefits you are eligible for. iHerb is an Equal Opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or veteran status. iHerb provides equal employment opportunities to all applicants for employment and prohibits discrimination and harassment. Show more [Apply now >](https://jobicy.com/jobs/147337-principal-engineer-ai-platform.md) * ![Upload CV](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI2NSIgaGVpZ2h0PSI2NSIgZmlsbD0ibm9uZSIgeG1sbnM6dj0iaHR0cHM6Ly92ZWN0YS5pby9uYW5vIj48ZyBjbGlwLXBhdGg9InVybCgjQSkiPjxwYXRoIGQ9Ik0wIDBINjVWNjVIMFYwWiIgZmlsbD0iIzAyOWFlYiIvPjxnIGZpbGw9IiNmZmYiIHN0cm9rZT0iI2ZmZiIgc3Ryb2tlLXdpZHRoPSIyIj48cGF0aCBkPSJNMzMuMDQ5IDE1LjQ1NGExLjQzIDEuNDMgMCAwIDAtMi4wOTcgMGwtNy41NzkgOC4xNDdhMS4zOCAxLjM4IDAgMCAwIC4wOSAxLjk3MyAxLjQ0IDEuNDQgMCAwIDAgMi4wMDgtLjA4OGw1LjEwOS01LjQ5MnYyMC42MWExLjQxIDEuNDEgMCAwIDAgMS40MjEgMS4zOTdjLjc4NSAwIDEuNDIxLS42MjUgMS40MjEtMS4zOTd2LTIwLjYxbDUuMTA5IDUuNDkyYTEuNDQgMS40NCAwIDAgMCAyLjAwOC4wODggMS4zOCAxLjM4IDAgMCAwIC4wOS0xLjk3M2wtNy41NzktOC4xNDZ6TTE2Ljc2OSAzOC40YzAtLjc3My0uNjItMS40LTEuMzg1LTEuNFMxNCAzNy42MjcgMTQgMzguNHYuMTAybC4yMTUgNi4yMjljLjIyMyAxLjY4LjcwMSAzLjA5NSAxLjgxMyA0LjIxOHMyLjUxIDEuNjA3IDQuMTcyIDEuODMzYzEuNi4yMTggMy42MzYuMjE4IDYuMTYuMjE4aDExLjI4bDYuMTYtLjIxOGMxLjY2Mi0uMjI2IDMuMDYxLS43MDkgNC4xNzItMS44MzNzMS41ODktMi41MzggMS44MTMtNC4yMThDNTAgNDMuMTEzIDUwIDQxLjA1NSA1MCAzOC41MDNWMzguNGMwLS43NzMtLjYyLTEuNC0xLjM4NS0xLjRzLTEuMzg1LjYyNy0xLjM4NSAxLjRsLS4xOSA1Ljk1OGMtLjE4MiAxLjM3LS41MTUgMi4wOTUtMS4wMjYgMi42MTJzLTEuMjI4Ljg1My0yLjU4MyAxLjAzOGMtMS4zOTUuMTktMy4yNDMuMTkzLTUuODkzLjE5M0gyNi40NjJjLTIuNjUgMC00LjQ5OC0uMDAzLTUuODkzLS4xOTMtMS4zNTUtLjE4NC0yLjA3Mi0uNTIxLTIuNTgzLTEuMDM4cy0uODQ0LTEuMjQyLTEuMDI2LTIuNjEyYy0uMTg3LTEuNDEtLjE5MS0zLjI3OS0uMTkxLTUuOTU4eiIvPjwvZz48L2c+PGRlZnM+PGNsaXBQYXRoIGlkPSJBIj48cGF0aCBmaWxsPSIjZmZmIiBkPSJNMCAwaDY1djY1SDB6Ii8+PC9jbGlwUGF0aD48L2RlZnM+PC9zdmc+) ### Upload your resume now To unlock remote work opportunities and be discovered by global employers. This job listing has been manually reviewed by the Jobicy Trust & Safety Team for compliance with our posting guidelines, including verification of the company's legitimacy, accuracy of job details, clarity of remote work policy, and absence of misleading or fraudulent content. ## How to apply ## See a few more Similar Software Engineering remote jobs * ![Mozilla logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2020/10/mozilla.jpg) Mozilla ### [Senior Software Engineer (Contract)](https://jobicy.com/jobs/145880-senior-software-engineer-contract.md) Are you passionate about Privacy Protections and the Open Web? Do you want to ship software to a large user base? Do you want your contributions to have a positive… ![Germany flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/de.svg) DE•ContractNEW* ![Boulevard logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2021/12/a45167e12b0ed51058a884cf9cd0c76f.png) Boulevard ### [Director of Engineering, Cloud & Reliability](https://jobicy.com/jobs/149111-director-of-engineering-cloud-reliability.md) Who is Boulevard? Boulevard provides the first and only client experience platform for appointment-based, self-care businesses. We empower our customers to give their clients more of the magical moments that… ![USA flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/us.svg) US•Full TimeUSD 209,500-270k/year* ![Upstart logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/bc2cfa2b-221-1.jpeg) Upstart ### [Software Engineer, Unsecured Installments](https://jobicy.com/jobs/149109-software-engineer-unsecured-installments.md) About Upstart At Upstart, we’re united by a mission that matters: to radically reduce the cost and complexity of borrowing for all Americans. Every day, we bring creativity, experimentation, and… ![USA flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/us.svg) US•Full TimeUSD 136k-196,600/year* ![Upstart logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/bc2cfa2b-221-1.jpeg) Upstart ### [Senior Software Engineer, Unsecured Installments](https://jobicy.com/jobs/149108-senior-software-engineer-unsecured-installments.md) About Upstart At Upstart, we’re united by a mission that matters: to radically reduce the cost and complexity of borrowing for all Americans. Every day, we bring creativity, experimentation, and… ![USA flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/us.svg) US•Full TimeUSD 160,400-223,900/year* ![Cresta logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2022/08/4d686e40b22cf2510b627c59afe5fd95.jpg) Cresta ### [Senior Software Engineer, Backend (AI Agent Integrations)](https://jobicy.com/jobs/149101-senior-software-engineer-backend-ai-agent-integrations.md) Cresta unlocks the true potential of the customer experience, turning every conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive… ![Canada flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/ca.svg) CA•Full TimeNEW* ![Grundfos logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/09/3a459ef1-221.png) Grundfos ### [Application Engineer – Digital Solutions](https://jobicy.com/jobs/149097-application-engineer-digital-solutions.md) Are you passionate about bridging the gap between cutting-edge digital technology and real-world customer success in water management? Then join us in our mission to pioneer smarter solutions for moving… ![Mexico flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/mx.svg) MX•Full TimeNEW* ![Zuora logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/43baa0bd-221.jpg) Zuora ### [Senior Customer Solution Engineer — AI Native/Consumption Quote to Cash](https://jobicy.com/jobs/149068-senior-customer-solution-engineer-ai-native-consumption-quote-to-cash.md) About ZuoraAt Zuora, we help businesses grow smarter and adapt faster. Our platform powers modern business models — from subscriptions and usage-based pricing to AI-driven and outcome-based offerings — helping… ![USA flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/us.svg) US•Full TimeUSD 136,800-177,100/year* ![Enode logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2026/06/70d7b7e3-221.webp) Enode ### [Software Engineer](https://jobicy.com/jobs/149062-software-engineer.md) About EnodeThe grid was built for predictable, centralised power. Today it’s absorbing intermittent supply from grid scale solar and wind, alongside dynamic demand from EVs, heat pumps, and home batteries…. ![Europe flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/eu.svg) ![Norway flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/no.svg) EU, NO•Full TimeJul 9* ![Ruby Labs logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/0799a7b7-221-1.jpeg) Ruby Labs ### [Lead IOS Engineer](https://jobicy.com/jobs/149055-lead-ios-engineer.md) About usRuby Labs is a leading tech company that creates and operates innovative consumer products. We offer a diverse range of opportunities across the health, education, and entertainment industries. Our… ![Europe flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/eu.svg) EU•Full TimeJul 9* ![Ruby Labs logo](https://jobicy.com/data/server-nyc0409/galaxy/mercury/2025/06/0799a7b7-221-1.jpeg) Ruby Labs ### [Lead Android Developer](https://jobicy.com/jobs/149054-lead-android-developer.md) About usRuby Labs is a leading tech company that creates and operates innovative consumer products. We offer a diverse range of opportunities across the health, education, and entertainment industries. Our… ![Europe flag](https://cloud.jobicy.com/nyc4-cold/img/round-flags/eu.svg) EU•Full TimeJul 9 [More Jobs](https://jobicy.com/jobs.md)