AI Product Analyst – Automation team

Remote from: Serbia
Annual salary: Undisclosed
Salary information is not provided for this position. Check our Salary Directory to estimate the average compensation for similar roles.
Department: Data Science & Analytics
Employment type: Full Time,
Job posted: 22 May 2026
Apply before: 22 Jun 2026
Experience level: Midweight
Views / Applies: 156 / 44

We are an online mental health treatment center aiming to make professional care accessible and affordable for everyone.

Mental Health Care
2019

AI Summary

MEDvidi is hiring a Product Analyst for its Automation team to own the full analytics stack for both AI-powered and conventional product features in a medical tech environment. The role involves analyzing production behavior of AI and automation workflows, investigating failures, and designing evaluations for AI quality. The analyst will work closely with Product, Engineering, Clinical, and other teams to support fast decisions under ambiguity. Key requirements include 3+ years of analytics experience, SQL/Python skills, and fluency in Russian and English.

Job Complexity

Easy Hard

AI Insight The role requires a blend of product analytics, AI evaluation, and operational analysis in a high-stakes medical domain, demanding strong technical skills and the ability to work in ambiguous environments.

Salary Analysis

AI Insight The job listing does not specify a salary range. For a Product Analyst with AI focus in the US, the market median is around $130,000. The offered salary is likely competitive given the specialized AI and medical domain requirements.

Key Skills

Product Analytics AI Evaluation SQL Python LLM Operational Analytics Data Dashboarding A/B Testing Medical Technology Automation

Cover Letter Sample

I am writing to express my strong interest in the AI Product Analyst position on the Automation team at MEDvidi. With over 3 years of experience in product analytics and a deep focus on AI evaluation, I am excited about the opportunity to drive data-informed decisions in a high-impact medical tech environment.

In my previous role, I designed and implemented analytics frameworks for AI-powered features, including hallucination detection and quality monitoring, which directly improved system reliability. My expertise in SQL and Python allows me to extract insights from complex production data, and I thrive in ambiguous, fast-paced settings where I can build ground truth from scratch.

I am particularly drawn to MEDvidi's mission of freeing clinicians from manual work through automation. I look forward to contributing to the team's success by turning production behavior into actionable evidence and ensuring AI quality remains a strategic function.

Thank you for considering my application. I am eager to discuss how my skills align with your needs.

Possible Interview Questions

Describe a time when you had to analyze the performance of an AI feature. How did you measure its success and identify failures?

I previously worked on an AI chatbot for customer support. I measured success by tracking resolution rate and user satisfaction, and identified failures by analyzing logs for repeated fallback triggers and sentiment drops. I set up dashboards to monitor these metrics in real-time and conducted root cause analysis for silent failures.

How would you design an experiment to compare an AI-generated medical note with a human-written one?

I would define key metrics like accuracy, completeness, and time savings. The experiment would involve random assignment of cases to AI or human, with clinicians rating the outputs blind. I'd also track downstream metrics like rework rates. Statistical significance would be assessed using appropriate tests, and I'd control for case complexity.

Explain a situation where you had to work with messy production data. How did you ensure data quality?

In a previous role, I worked with logs that had inconsistent formatting and missing timestamps. I built a data pipeline that included validation steps, such as checking for nulls and outliers, and used Python to clean and standardize the data. I also collaborated with engineering to improve logging practices.

How do you prioritize which metrics to track for a new automation feature?

I start by understanding the business goals and user needs. For automation, key metrics include success rate, time saved, error rate, and user adoption. I also consider operational impact like reduced manual workload. I align with stakeholders to ensure metrics are actionable and tied to decision-making.

Describe a time you had to communicate a complex analytical finding to a non-technical audience. How did you ensure understanding?

I once analyzed the impact of a new algorithm on customer churn. I created a simple visual showing churn rates before and after, and used analogies to explain the statistical significance. I focused on the business implications rather than technical details, and invited questions to clarify.

Description

At MEDvidi, the Automation team builds and operates a mix of AI and conventional product features that automate clinical and operational workflows – from AI-drafted medical documentation to automated resolution of patient issues, intake verification flows, provider tooling, and the backend pipelines behind them. We free clinicians and operations from manual work a machine can do reliably and safely, in a domain where mistakes have real clinical and compliance consequences.

It’s a full-stack product team, not an AI lab: we ship UI, backend workflows, integrations, and AI features side by side, and we measure them the same way.

The team works closely with Product, Engineering, MedOps, Clinical people. We’re hiring an Product Analyst to own analytics for the team end-to-end as our automated surface area grows.

The role

You’ll be the Product Analyst for the team – across everything we ship, both classical product features and AI-powered ones. You’ll turn the production behavior of our systems into evidence the team can act on: which parts work, where they fail silently, and what to fix next. The work blends product analytics, AI evaluation, and operational analysis.

Responsibilities

Own the full product analytics stack for features the team ships – discovery and sizing, experiment design and analysis, funnel and adoption analysis, post-release impact assessment – across both AI-powered and conventional product features.
Analyze production behavior of AI and automation workflows across clinical and operational journeys.
Investigate hallucinations, silent failures, and quality degradation in AI-generated output.
Work directly with logs, transcripts, generated artifacts, and operational data to find patterns dashboards miss.
Design evaluations for AI quality – both automated and human review – and close the loop into prompt and product changes.
Build monitoring and dashboards for AI and automation health, and the operational metrics they roll up to.
Measure business and operational impact of automation initiatives and surface the risks they introduce.
Partner closely with Product, Engineering, Clinical, MedOps, and Legal to support fast decisions under ambiguity.

Requirements

3+ years of experience in Product Analytics/Operational Analytics/Data Analytics, or analytics roles within complex production systems.
Hands-on experience evaluating LLM or AI output – prompt iteration, eval frameworks and human review pipelines is a strong plus.
Effective, structured written and verbal communication – able to turn analysis into crisp, decision-ready writeups that non-analysts can act on.
Comfort in ambiguous, fast-changing environments where ground truth often has to be constructed.
Strong product / operational analytics background, comfortable owning metrics end-to-end and writing SQL and Python code against messy production data.
Fluent Russian; English level B2 or higher,
Ability to work CET-overlapping hours.

What we offer

Work on a modern AI product where quality is a strategic function, not a formality
Real influence on product direction and engineering practices
High-impact work in a product-driven medical tech company
Startup environment: fast iteration, high ownership, minimal bureaucracy
Competitive compensation and growth opportunities

Benefits