Evaluation Scenario Writer - AI Agent Testing Specialist
MindriftPutrajaya, Putrajaya, Malaysia2 hari lalu
Penerangan pekerjaanBachelor’s and / or Master’s degree in Computer Science, Software Engineering, Data Science / Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / NLP, Information Systems or related fields Background in QA, software testing, data analysis, or NLP annotation Good understanding of test design principles (e.g., reproducibility, coverage, edge cases) Strong written communication skills in English Comfortable with structured formats like JSON / YAML for scenario description Can define expected agent behaviors (gold paths) and scoring logic Basic experience with Python and JavaScript Curious and open to working with AI‑generated content, agent logs, and prompt‑based behavior Ready to learn new methods, able to switch between tasks and topics quickly, and sometimes work with challenging, complex guidelines Fully remote freelance role – only requires a laptop, internet connection, available time, and enthusiasm to take on a challenge Experience in writing manual or automated test cases Familiarity with LLM capabilities and typical failure modes Understanding of scoring metrics (precision, recall, coverage, reward functions) Get paid for your expertise, with rates up to $38 / hr depending on your skills, experience, and project needs Participate in a flexible, remote, freelance project that fits around your primary professional or academic commitments Gain valuable experience to enhance your portfolio through an advanced AI project Influence how future AI models understand and communicate in your field of expertise
Mindrift is looking for a freelance Agent Scenarios Designer based in the specified country. The role focuses on designing realistic and structured evaluation scenarios for LLM‑based agents, testing agent outputs, and refining tests. You will work on a flexible schedule and receive pay up to $38 / hr based on experience.
What We Do
The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting‑edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real‑world expertise from across the globe.
About the Role
You will design realistic and structured evaluation scenarios, create test cases that simulate human‑performed tasks, and define gold‑standard behavior to compare agent actions against. Your work will ensure each scenario is clearly defined, well‑scored, and easy to execute and reuse. You need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.
Responsibilities
- Design structured test scenarios based on real‑world tasks
- Define the golden path and acceptable agent behavior
- Annotate task steps, expected outputs, and edge cases
- Work with developers to test scenarios and improve clarity
- Review agent outputs and adapt tests accordingly
How to Get Started
Apply to this posting, qualify, and you’ll have the chance to contribute to projects aligned with your skills on your own schedule. From creating training prompts to refining model responses, you’ll help shape the future of AI while ensuring technology benefits everyone.
Requirements
Nice to Have
Benefits
#J-18808-Ljbffr
Buat amaran kerja untuk carian ini
Evaluation Writer Ai • Putrajaya, Putrajaya, Malaysia
Pekerjaan berkaitan
Position : Content Writer ( VLLM).Tenure : 1 year contract ( Renewable basis).Salary : RM 3200- RM 3700 ( Subject to Experience).
Location : Kuala Lumpur / Petaling Jaya.You will analyze images and write...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
This role is to lead initiatives focused on client acquisition and AI-enhanced user experiences in our global mobile trading platform.
This role is pivotal in driving growth by enhancing user onboar...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
%20Sdn%20Bhd)
Create and present contents via Youtube Platform.Familiar with Financial Services industry (not limited to fintech, loans, BPL).
Work hours 9am-5pm, Monday - Friday.Hiring Immediately - Permanant an...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
- Baharu!
Gen AI Developer Sr Consultant - Marketing, Commerce & Product - PH PDC
Deloitte PLTKuala Lumpur, Kuala Lumpur, Malaysia Select how often (in days) to receive an alert : .Gen AI Developer Sr Consultant - Marketing, Commerce & Product - PH PDC.
Title : Full Stack Senior Consultant based in Deloitte Consulting Philippines ...Tunjukkan lagiKemas kini terakhir: 16 jam yang lalu
- Dinaikkan pangkat
Search Engine Optimization (SEO) Specialist
DigitalKlang City, Selangor, Malaysia Search Engine Optimization (SEO) Specialist.Hire Digital is looking for an experienced.Senior SEO Specialist (Freelance, Remote).
This role requires a strategic thinker who can drive end-to-end SEO ...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
Senior AI Research Engineer, Model Inference (100% Remote)
Tether Operations LimitedPuchong, 10, MY Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution.
Our cutting-edge solutions empower businesses—from ex...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
Junior Content Writer
EPS ConsultantsPetaling Jaya, Selangor, MYQuick Apply
- Dinaikkan pangkat
Instructor for AI-Assisted Programming Workshops (part-time)
TripleTenShah Alam, Shah Alam, Malaysia TripleTen Kuala Lumpur, Federal Territory of Kuala Lumpur, Malaysia.Instructor – AI Course Trainer.We’re building an online course for experienced software developers on the practical use of Cursor...Tunjukkan lagiKemas kini terakhir: 11 hari yang lalu
- Dinaikkan pangkat
- Baharu!
Data Analytics Specialist - Associate (Assurance - Data Intelligence Delivery)
Ernst & YoungKuala Lumpur, Kuala Lumpur, Malaysia EY's commitment to the quality and integrity of our audits is exemplified by our global audit methodology and our thorough quality controls that are applied to every client engagement.Together with...Tunjukkan lagiKemas kini terakhir: 16 jam yang lalu
Lead Technical Author
Talent SwitchKuala Lumpur, Malaysia Lead the investigation, evaluation and implementation of next generation technologies, especially generative AI, to help drive productivity and quality gains in technical authoring.Define and execu...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
Product Manager - Growth and AI
Flintex Consulting Pte LtdKuala Lumpur, 14, myQuick Apply
AI BUSINESS ANALYST
Talent SwitchKuala Lumpur, Malaysia Work with business stakeholders to identify and define use cases for AI solutions (e.Analyze current workflows and systems to determine where AI can enhance productivity, accuracy, or decision-maki...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
AI Data Specialist - Chinese
RWS GroupKuala Lumpur, Kuala Lumpur, Malaysia AI Data Specialist - Chinese (Remote).We are looking for an AI Data Specialist to support the improvement of AI-generated content in English.
This is a freelance, part‑time role based remotely with ...Tunjukkan lagiKemas kini terakhir: 18 hari yang lalu
- Dinaikkan pangkat
SENIOR QA ENGINEER
Teknion Furniture Systems (M) Sdn BhdKlang City, Selangor, Malaysia Assist in developing, leading and executing Quality Management strategies.Participate in ensuring consistent product quality by developing and enforcing good manufacturing practices systems, valida...Tunjukkan lagiKemas kini terakhir: 17 hari yang lalu
- Dinaikkan pangkat
Italian Voice Actor - AI Trainer
Invisible Expert MarketplacePort Klang, Port Klang, Malaysia Italian Voice Acting Specialist – AI Trainer.We’re looking for a highly skilled Italian voice acting professional to help build AI voice models.
You’ll use cutting‑edge tools, record and evaluate It...Tunjukkan lagiKemas kini terakhir: 9 hari yang lalu
- Dinaikkan pangkat
AI Engineer (Multi-agent system)
Hiredly XKuala Lumpur, Kuala Lumpur, Malaysia As an AI Engineer (Entry to Mid level), you’ll play a hands-on role in the development of our client's multi-agent AI systems.
You’ll implement prompt templates, integrate LLMs, build user-facing fe...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
- Baharu!
Content Writer
SHUTTERSTUDIOCyberjaya, Selangor, Malaysia Develop, organize, and manage all published content (images, videos, writing) across all social media platforms.Ensure clients' brand voice, messaging, and visual design consistency across all soci...Tunjukkan lagiKemas kini terakhir: 18 jam yang lalu
- Dinaikkan pangkat
Ads Relevance Specialist - AI Data Service and Operations
TikTokKuala Lumpur, Kuala Lumpur, Malaysia Ads Relevance Specialist - AI Data Service and Operations.TikTok, Kuala Lumpur, Federal Territory of Kuala Lumpur, Malaysia.
About the team : Our Search Operations team supports our efforts to addres...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
Senior Research Engineer Multimodal & Video Foundation Model (100% Remote)
Tether Operations LimitedPuchong, 10, MY Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution.
Our cutting-edge solutions empower businesses—from ex...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
Content Writer
Two95 International Inc.Setia Alam, Selangor, MYQuick Apply
- Dinaikkan pangkat
(WFH) SEO Content Writer
Alpha Iota BPO Sdn BhdKuala Lumpur, Malaysia Join Our Alpha Iota Family, Where Everyone Wins!.Exciting Work-from-Home Opportunities.Learning & Development Programs to Upskill Yourself.
Health and Wellness Perks & Benefits.Motivating and Suppor...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu