QA Automation Engineer

Remote from: Poland
Annual salary: Undisclosed
Salary information is not provided for this position. Check our Salary Directory to estimate the average compensation for similar roles.
Department: Software Engineering
Employment type: Full Time,
Job posted: 22 Oct 2025
Apply before: 22 Nov 2025
Experience level: Senior
Views / Applies: 61 / 3

Helping organizations adopt software innovations and market-leading solutions through expert software development services.

Altoros – an established provider of software development services, is looking for a QA Automation Engineer.

Requirements

Responsibilities & Responsibilities:

It’s a part-time position.

Own the test strategy across FE, BE, and LLM/agent workflows; define the test pyramid and coverage goals.
Build and maintain automation for:

Design golden datasets and rubrics for LLM output evaluation (pass/fail thresholds, human‑in‑the‑loop review).
Set up regression/eval harnesses for prompts, tools, and agents; track drift and hallucination rates.
Implement guardrails and adversarial/red‑team tests for safety, privacy, bias, and toxicity.

Non‑functional testing: baseline and monitor performance, reliability, and resilience for key user journeys.
CI/CD ownership: integrate tests into pipelines, gate releases, reduce flakiness, and surface quality KPIs (e.g., failure rate, MTTR, coverage).
Test data & environments: manage fixtures, synthetic data, and stable seeds for deterministic runs.
Quality operations: triage defects, drive root‑cause analysis, and keep crisp, proactive communication with PM, Design, and Engineering.
Documentation: maintain clear test plans, playbooks, and release criteria.

Nice to have:

Experience with agent frameworks, tool calling, retrieval/RAG, or tracing.
Familiarity with evaluation/observability tools for LLM apps (e.g., prompt testing frameworks, tracing dashboards).
Performance testing (e.g., k6/JMeter) and basic security/safety testing practices.

Join Altoros Team!

Apply now >