Evaluation Scenario Writer - AI Agent Testing SpecialistMindrift • Muar, Johor, Malaysia

Evaluation Scenario Writer - AI Agent Testing Specialist

Mindrift • Muar, Johor, Malaysia

1 hari lalu

Penerangan pekerjaan

Mindrift is looking for a freelance Agent Scenarios Designer based in the specified country. The role focuses on designing realistic and structured evaluation scenarios for LLM‑based agents, testing agent outputs, and refining tests. You will work on a flexible schedule and receive pay up to $38 / hr based on experience.

What We Do

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting‑edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real‑world expertise from across the globe.

About the Role

You will design realistic and structured evaluation scenarios, create test cases that simulate human‑performed tasks, and define gold‑standard behavior to compare agent actions against. Your work will ensure each scenario is clearly defined, well‑scored, and easy to execute and reuse. You need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Responsibilities

Design structured test scenarios based on real‑world tasks
Define the golden path and acceptable agent behavior
Annotate task steps, expected outputs, and edge cases
Work with developers to test scenarios and improve clarity
Review agent outputs and adapt tests accordingly

How to Get Started

Apply to this posting, qualify, and you’ll have the chance to contribute to projects aligned with your skills on your own schedule. From creating training prompts to refining model responses, you’ll help shape the future of AI while ensuring technology benefits everyone.

Requirements

Bachelor’s and / or Master’s degree in Computer Science, Software Engineering, Data Science / Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / NLP, Information Systems or related fields

Background in QA, software testing, data analysis, or NLP annotation

Good understanding of test design principles (e.g., reproducibility, coverage, edge cases)

Strong written communication skills in English

Comfortable with structured formats like JSON / YAML for scenario description

Can define expected agent behaviors (gold paths) and scoring logic

Basic experience with Python and JavaScript

Curious and open to working with AI‑generated content, agent logs, and prompt‑based behavior

Ready to learn new methods, able to switch between tasks and topics quickly, and sometimes work with challenging, complex guidelines

Fully remote freelance role – only requires a laptop, internet connection, available time, and enthusiasm to take on a challenge

Nice to Have

Experience in writing manual or automated test cases

Familiarity with LLM capabilities and typical failure modes

Understanding of scoring metrics (precision, recall, coverage, reward functions)

Benefits

Get paid for your expertise, with rates up to $38 / hr depending on your skills, experience, and project needs

Participate in a flexible, remote, freelance project that fits around your primary professional or academic commitments

Gain valuable experience to enhance your portfolio through an advanced AI project

Influence how future AI models understand and communicate in your field of expertise

#J-18808-Ljbffr

Buat amaran kerja untuk carian ini

Evaluation Writer Ai • Muar, Johor, Malaysia

Pekerjaan berkaitan

Interpreter - Cantonese - Work-From-Home (Australia)

Teleperformance • Muar, Johor, Malaysia

Excellent listening, retention and note taking skills to maintain a high level of accuracy.Ability to concentrate, stay detached from the conversation, and always remain neutral and objective.Abili...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Talent Acquisition Specialist

Flowmingo AI • Muar, Johor, Malaysia

Talent Acquisition Specialist – A role in Flowmingo Partner Program.Flowmingo is a Y Combinator‑backed AI platform that's redefining how companies hire. We’re an AI‑powered interview platform helpin...Tunjukkan lagi

Kemas kini terakhir: 29 hari yang lalu • Dinaikkan pangkat

Enrichment Manager - Scenic Cruises

Scenic - Luxury Cruises & Tours • Batu Pahat, Johor, Malaysia

Amazing People Creating Wonder.Thank you for considering Scenic Group.Location : Europe - multiple countries.Experience in the field of entertainment or as an animator at a hotel or on a cruise ship...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Medical Expert (SME)

Invisible Expert Marketplace • Batu Pahat, Johor, Malaysia

Medical Expert (SME) – Consultant Role Description.We are seeking a highly experienced Medical Doctor to serve as a consultant on AI training data projects for leading AI model builders and enterpr...Tunjukkan lagi

Kemas kini terakhir: 26 hari yang lalu • Dinaikkan pangkat

Wolof Language Specialist - AI Trainer

Invisible Expert Marketplace • Muar, Johor, Malaysia

Wolof Language Specialist – AI Trainer.AI models for Wolof speakers worldwide.Review and annotate Wolof content for training datasets. Evaluate AI-generated outputs for accuracy, fluency, and cultur...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Freelance Electrical Engineer - AI Trainer

Mindrift • Batu Pahat, Johor, Malaysia

This opportunity is limited to candidates residing in the specified country.Location may affect eligibility and rates.Please submit your resume in English and indicate your level of English.At Mind...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Senior Technical Support Engineer - Cybersecurity SaaS

Intuition Machines • Batu Pahat, Johor, Malaysia

Senior Technical Support Engineer - Cybersecurity SaaS.Be among the first 25 applicants.Intuition Machines uses AI / ML to build enterprise security products. We apply our research to systems that ser...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Real Estate Agent

MegaHarta Real Estate • Muar, Johor, Malaysia

MegaHarta Real Estate Group, established in March 2002, is a leading property agency based in Kuala Lumpur and Petaling Jaya, Malaysia. As a licensed agency registered with The Board of Valuers, App...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Principal Engineer Test Technology and Innovation

Infineon Technologies • Malacca City, Malacca, Malaysia

Principal Engineer Test Technology and Innovation.Principal Engineer Test Technology and Innovation.This role involves defining and developing optimal test solutions and handling capabilities for n...Tunjukkan lagi

Kemas kini terakhir: 1 hari yang lalu • Dinaikkan pangkat

Web Scraper / Data Engineer

STERRY • Malacca City, Malacca, Malaysia

Be among the first 25 applicants.At STERRY, we’re not your average Growth Marketing Agency—we’re the rocket fuel behind crowdfunding and e-commerce success. Since day one, we’ve helped clients pull ...Tunjukkan lagi

Kemas kini terakhir: 5 hari yang lalu • Dinaikkan pangkat

Accounting Consultant (12-month contract, Full-Time, Remote)

Animoca Brands • Muar, Johor, Malaysia

Animoca Brands Corporation Limited (ACN : ) is a Web3 leader that leverages tokenization and blockchain to deliver digital property rights to consumers, helping to establish the open metaverse and ...Tunjukkan lagi

Kemas kini terakhir: 10 hari yang lalu • Dinaikkan pangkat

Audit Manager (Johor Bahru)

ACCA Careers • Kebun Baharu, Johor, Malaysia

Be among the first 25 applicants.In today’s world, audit is no longer just about compliance.It is about cutting edge and risk based methodologies and knowing what is happening in the outside world....Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Freelance Ilocano On-Demand Interpreter

THE LANGUAGE GROUP, LLC • Batu Pahat, Johor, Malaysia

Accurately interprets messages between non-English speaking patients and healthcare providers or support personnel on-site. Adheres to the National Standards of Practice for Interpreters in Health C...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Become a Freelance Merchandising Auditor, Bintulu, Malaysia

CXG group • Muar, Johor, Malaysia

Turn your eye for detail into a.We are looking for Freelance Merchandising Auditors in Tawau, Malaysia, to evaluate store conditions and device functionality for a world-renowned tech brand.As a Me...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Voice Talent Required - Malay

LILT AI • Batu Pahat, Johor, Malaysia

Join to apply for the Voice Talent Required - Malay role at LILT AI.AI is changing how the world communicates — and LILT is leading that transformation. We're on a mission to make the world's inform...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Teach English From Home with YiYi!

YiYiEnglish • Batu Pahat, Johor, Malaysia

Stable Class Hours = Stable Income.Fixed Student & Teacher pairing.Long-term employment only! (1 year+).Since 2017, YiYiEnglish has been providing high quality English lessons to students in China....Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat

Talent Acquisition Specialist - Contractor

Bybit • Muar, Johor, Malaysia

Talent Acquisition Specialist - Contractor.Location : Worldwide Remote for 6-12 months contractor.Established in March 2018, Bybit is one of the fastest growing cryptocurrency derivatives exchanges,...Tunjukkan lagi

Kemas kini terakhir: 19 hari yang lalu • Dinaikkan pangkat

Technical Author (multiple roles and seniority levels)

Canonical • Batu Pahat, Johor, Malaysia

Technical Author (multiple roles and seniority levels).Canonical Kuala Lumpur, Federal Territory of Kuala Lumpur, Malaysia. Join or sign in to find your next job.Technical Author (multiple roles and...Tunjukkan lagi

Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat