Evaluation Scenario Writer - AI Agent Testing SpecialistMindrift • Ipoh, Perak, Malaysia

Evaluation Scenario Writer - AI Agent Testing Specialist

Mindrift • Ipoh, Perak, Malaysia

7 jam yang lalu

Penerangan pekerjaan

Mindrift is looking for a freelance Agent Scenarios Designer based in the specified country. The role focuses on designing realistic and structured evaluation scenarios for LLM‑based agents, testing agent outputs, and refining tests. You will work on a flexible schedule and receive pay up to $38 / hr based on experience.

What We Do

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting‑edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real‑world expertise from across the globe.

About the Role

You will design realistic and structured evaluation scenarios, create test cases that simulate human‑performed tasks, and define gold‑standard behavior to compare agent actions against. Your work will ensure each scenario is clearly defined, well‑scored, and easy to execute and reuse. You need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Responsibilities

Design structured test scenarios based on real‑world tasks
Define the golden path and acceptable agent behavior
Annotate task steps, expected outputs, and edge cases
Work with developers to test scenarios and improve clarity
Review agent outputs and adapt tests accordingly

How to Get Started

Apply to this posting, qualify, and you’ll have the chance to contribute to projects aligned with your skills on your own schedule. From creating training prompts to refining model responses, you’ll help shape the future of AI while ensuring technology benefits everyone.

Requirements

Bachelor’s and / or Master’s degree in Computer Science, Software Engineering, Data Science / Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / NLP, Information Systems or related fields

Background in QA, software testing, data analysis, or NLP annotation

Good understanding of test design principles (e.g., reproducibility, coverage, edge cases)

Strong written communication skills in English

Comfortable with structured formats like JSON / YAML for scenario description

Can define expected agent behaviors (gold paths) and scoring logic

Basic experience with Python and JavaScript

Curious and open to working with AI‑generated content, agent logs, and prompt‑based behavior

Ready to learn new methods, able to switch between tasks and topics quickly, and sometimes work with challenging, complex guidelines

Fully remote freelance role – only requires a laptop, internet connection, available time, and enthusiasm to take on a challenge

Nice to Have

Experience in writing manual or automated test cases

Familiarity with LLM capabilities and typical failure modes

Understanding of scoring metrics (precision, recall, coverage, reward functions)

Benefits

Get paid for your expertise, with rates up to $38 / hr depending on your skills, experience, and project needs

Participate in a flexible, remote, freelance project that fits around your primary professional or academic commitments

Gain valuable experience to enhance your portfolio through an advanced AI project

Influence how future AI models understand and communicate in your field of expertise

#J-18808-Ljbffr

Buat amaran kerja untuk carian ini

Evaluation Writer Ai • Ipoh, Perak, Malaysia