Freelance Agent Evaluation Engineer

Remote from: LATAM
Salary, yearly, USD: 80,000
Department: Software Engineering
Employment type: Full Time,
Job posted: 13 Jul 2026
Apply before: 13 Aug 2026
Views / Applies: 29 / 2

Mindrift connects AI experts and clients to advance Generative AI models.

AI Summary

This role involves creating challenging evaluation tasks for AI coding agents by building realistic developer environments and writing rigorous test suites. You will design tasks from intermediate states of simulated codebases, ensuring they are solvable by AI agents while tests accept all valid solutions. The position requires 5+ years of software development experience with Python, JavaScript/TypeScript, Docker, Postgres, Kafka, and Redis, along with strong test-writing skills. It is a project-based freelance opportunity with flexible scheduling, and tasks are estimated at 20 hours each with compensation up to $40/hour equivalent.

Role DNA

Job Complexity

Easy Hard

Pace & Pressure

Relaxed Fast-paced

Autonomy Level

Guided Full Ownership

Communication Load

Independent Highly Collaborative

AI Insight Creating tasks that genuinely challenge state-of-the-art AI coding models requires deep understanding of model failure modes and non-trivial test design, making it exceptionally hard.

Salary Analysis

AI Insight The offered salary of up to $80,000 per year (based on $40/hr) is below the typical US market range for senior software engineers ($90k-$160k) but is common for project-based freelance roles with flexible scheduling.

Core Skills Required

Python JavaScript TypeScript React Docker Postgres Kafka Redis AI Evaluation Test Writing

Cover Letter Sample

Dear Hiring Team,

I am excited to apply for the Freelance Agent Evaluation Engineer position. With over 5 years of software development experience and a strong background in Python, JavaScript/TypeScript, and Docker, I am well-prepared to design challenging evaluation tasks for AI coding agents. My experience writing comprehensive functional and integration tests ensures I can create robust verification criteria that accept all valid solutions while rejecting incorrect ones. I am drawn to the innovative nature of this role and the opportunity to contribute to advancing AI systems. Thank you for considering my application.

Sample Interview Questions

Describe your experience designing evaluation tasks for AI or complex software systems.

I have designed evaluation benchmarks for a machine learning platform where I created diverse test scenarios that covered edge cases. I focused on ensuring tasks were neither too simple nor ambiguous, and I iteratively refined them based on model performance and feedback.

How would you create a test that accepts multiple valid solutions for an open-ended coding task?

I would define the core requirements and use property-based testing where possible, focusing on output invariants. For example, if the task is to implement a sorting algorithm, the test should verify the output is sorted and stable rather than matching a specific implementation.

Can you give an example of a tricky scenario where an AI agent might solve a task incorrectly but still pass a naive test?

One example is when an agent overfits to common patterns, like hardcoding values instead of generalizing. A naive test might only check the provided sample inputs, while a robust test would include unseen variations and require the solution to handle them.

How do you approach debugging and refining a task that an AI agent fails unexpectedly?

I analyze the agent's failure mode to understand if the task is ambiguous, too difficult, or if the test criteria are too strict. I then adjust the prompt or the test to better isolate the desired behavior while ensuring fairness.

Explain your process for building a realistic simulated development environment for evaluation.

I start by defining a plausible company context with a codebase, documentation, and a mix of tickets and bugs. I then set up realistic infrastructure like Docker containers and incorporate partial solutions to create intermediate states. The goal is to mimic a real developer's workflow.

Please submit your CV in English and indicate your level of English proficiency.

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

What this opportunity involves

We’re building a dataset to evaluate AI coding agents – how well a model handles real-world developer tasks.

You’ll create challenging tasks and evaluation criteria within realistic simulated environments:

Build realistic developer environments – a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
Design tasks from intermediate states of these environments – craft the prompt, define what “solved” means, and ensure the task is solvable by an AI agent
Write tests that verify agent solutions – accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
Iterate on tasks and tests based on QA feedback – review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

Not data labeling
Not prompt engineering
Not writing code from scratch – the agent writes most of the code; you guide and evaluate

What we look for

5+ years in software development
Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
Experience writing tests (functional, integration)
English proficiency – B2+

Why this is hard

Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions – writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Effort estimate

Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation

Up to $40/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.

Apply now >