Evaluation Scenario Writer - AI Agent Testing Specialist
MindriftCentral George Town, Penang, Malaysia6 hari lalu
Penerangan pekerjaanBachelor’s and / or Master’s degree in Computer Science, Software Engineering, Data Science / Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / NLP, Information Systems or related fields Background in QA, software testing, data analysis, or NLP annotation Good understanding of test design principles (e.g., reproducibility, coverage, edge cases) Strong written communication skills in English Comfortable with structured formats like JSON / YAML for scenario description Can define expected agent behaviors (gold paths) and scoring logic Basic experience with Python and JavaScript Curious and open to working with AI‑generated content, agent logs, and prompt‑based behavior Ready to learn new methods, able to switch between tasks and topics quickly, and sometimes work with challenging, complex guidelines Fully remote freelance role – only requires a laptop, internet connection, available time, and enthusiasm to take on a challenge Experience in writing manual or automated test cases Familiarity with LLM capabilities and typical failure modes Understanding of scoring metrics (precision, recall, coverage, reward functions) Get paid for your expertise, with rates up to $38 / hr depending on your skills, experience, and project needs Participate in a flexible, remote, freelance project that fits around your primary professional or academic commitments Gain valuable experience to enhance your portfolio through an advanced AI project Influence how future AI models understand and communicate in your field of expertise
Mindrift is looking for a freelance Agent Scenarios Designer based in the specified country. The role focuses on designing realistic and structured evaluation scenarios for LLM‑based agents, testing agent outputs, and refining tests. You will work on a flexible schedule and receive pay up to $38 / hr based on experience.
What We Do
The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting‑edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real‑world expertise from across the globe.
About the Role
You will design realistic and structured evaluation scenarios, create test cases that simulate human‑performed tasks, and define gold‑standard behavior to compare agent actions against. Your work will ensure each scenario is clearly defined, well‑scored, and easy to execute and reuse. You need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.
Responsibilities
- Design structured test scenarios based on real‑world tasks
- Define the golden path and acceptable agent behavior
- Annotate task steps, expected outputs, and edge cases
- Work with developers to test scenarios and improve clarity
- Review agent outputs and adapt tests accordingly
How to Get Started
Apply to this posting, qualify, and you’ll have the chance to contribute to projects aligned with your skills on your own schedule. From creating training prompts to refining model responses, you’ll help shape the future of AI while ensuring technology benefits everyone.
Requirements
Nice to Have
Benefits
#J-18808-Ljbffr
Buat amaran kerja untuk carian ini
Evaluation Writer Ai • Central George Town, Penang, Malaysia
Pekerjaan berkaitan%20SDN%20BHD)
%20Sdn%20Bhd)
- Dinaikkan pangkat
Content Creator
DK UKTIMATE (M) SDN BHDSungai Petani, Kedah, Malaysia Diploma in Marketing, Communications, or a related field (or equivalent experience).Proven experience creating content across different platforms (social media, blogs, websites, video, etc.Strong w...Tunjukkan lagiKemas kini terakhir: 15 hari yang lalu
- Dinaikkan pangkat
Azure Architect (AI Adoption / Security)
Softenger (Malaysia) Sdn BhdSeberang Perai, Penang, Malaysia Job Title : AI Architect (Adoption / Security).We are hiring for key roles to support a major enterprise‑scale AI transformation program.
Candidates will work closely with business and IT teams to driv...Tunjukkan lagiKemas kini terakhir: 2 hari yang lalu
- Dinaikkan pangkat
Project Manager - Remote, Cantonese Speaker ( Mobile Apps / Web / AI Solution)
REDSO INNOVATION SDN. BHD.Kulim, Kedah, Malaysia This is an exciting opportunity to join REDSO INNOVATION SDN.Project Manager (Mobile Apps / Web / AI Solution).In this full-time fully remote role, you will be responsible for leading the successfu...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
Senior AI Research Engineer, Model Inference (100% Remote)
Tether Operations LimitedMalacca, 04, MY Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution.
Our cutting-edge solutions empower businesses—from ex...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
Senior Specialist / Specialist - Test (Impedance Measurement)
AT&SKulim, Kedah, Malaysia Senior Specialist / Specialist - Test (Impedance Measurement).Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.
IC substrates and printed circuit boards....Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
Become a Luxury Brand Evaluator Automobile Project in Penang, Malaysia
CXGGeorge Town, Penang, Malaysia Are you a luxury automobile enthusiast who appreciates the finer details of high-end vehicles? If the answer is yes, we are looking for you! As a Luxury Brand Evaluator, you will step into the worl...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
Nahuatl Language Expert - AI Trainer
Invisible Expert MarketplaceBayan Lepas, Penang, Malaysia Nahuatl Language Expert – AI Trainer.We’re looking for Nahuatl language specialists who live and breathe regional dialects, phonetics, syntax, idiomatic expressions, oral traditions, and cultural r...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
- Baharu!
AI-Powered Market Outreach SDR (Hybrid)
S&P GlobalGeorge Town, Penang, Malaysia A leading financial information provider in George Town is looking for an Associate in Market Outreach to drive sales development initiatives.
This role involves managing inbound inquiries, making c...Tunjukkan lagiKemas kini terakhir: 11 jam yang lalu
- Dinaikkan pangkat
AI Data Specialist - Chinese
RWS GroupCentral George Town, Penang, Malaysia AI Data Specialist - Chinese (Remote).We are looking for an AI Data Specialist to support the improvement of AI-generated content in English.
This is a freelance, part‑time role based remotely with ...Tunjukkan lagiKemas kini terakhir: 24 hari yang lalu
Senior Research Engineer Multimodal & Video Foundation Model (100% Remote)
Tether Operations LimitedMalacca, 04, MY Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution.
Our cutting-edge solutions empower businesses—from ex...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
- Baharu!
Credit Evaluation & Approval Specialist
AFFIN GroupPenangMalaysia, Penang, Malaysia A leading financial institution in Malaysia, Penang, is seeking an individual to join their team for credit evaluation and document verification.
The ideal candidate will assess credit proposals, en...Tunjukkan lagiKemas kini terakhir: 11 jam yang lalu
- Dinaikkan pangkat
- Baharu!
AI Data Specialist - Chinese Simplified
RWS TrainAISungai Petani, Kedah, Malaysia Part-time - 10+ hours per week.Flexible - work whenever you want.Until the end of December 2025 (an extension is possible).
Are you a student, recent graduate, stay‑at‑home parent, gig worker, or pr...Tunjukkan lagiKemas kini terakhir: 11 jam yang lalu
- Dinaikkan pangkat
Content Creator Specialist - AI Trainer
Invisible Expert MarketplaceBatu Kawan, Penang, Malaysia Content Creator Specialist - AI Trainer.Be among the first 25 applicants.Are you a creative content creator eager to shape the future of AI?.
Large-scale language and vision models are transforming ...Tunjukkan lagiKemas kini terakhir: 10 hari yang lalu
- Dinaikkan pangkat
Associate Research Analyst
Informa TechTargetGeorge Town, Penang, Malaysia Do you want to develop your career and make an impact in the fast-growth, fast-moving B2B technology space?.At Informa TechTarget, you’ll collaborate and grow alongside some of the industry’s most ...Tunjukkan lagiKemas kini terakhir: 30+ hari yang lalu
- Dinaikkan pangkat
Remote SEO Content Writer for Growth and Impact
AlphaiotabpoCentral George Town, Penang, Malaysia A dynamic digital marketing firm based in Kuala Lumpur is seeking a content creator with expertise in social media advertising.
The role involves researching, writing, and optimizing content while a...Tunjukkan lagiKemas kini terakhir: 3 hari yang lalu
- Dinaikkan pangkat
AI Evaluator - Cantonese (Chinese) - Malaysia
CrowdGen by AppenSungai Petani, Kedah, Malaysia Join CrowdGen as we launch an exciting new AI Voice Interaction Project designed to help improve the way voice assistants understand and respond to users! We’re looking for detail-oriented contribu...Tunjukkan lagiKemas kini terakhir: 6 hari yang lalu
- Dinaikkan pangkat
Freelance Medical Content Writer
DigitalBayan Lepas, Penang, Malaysia Hire Digital is seeking a remote.Freelance Medical Content Writer.The ideal candidate has a solid background in medical or pharmaceutical writing and can translate complex topics into clear and eng...Tunjukkan lagiKemas kini terakhir: 6 hari yang lalu
- Dinaikkan pangkat
T&T Graduate Analyst - Artificial Intelligence & Data - MY - Sep 2025
Deloitte PLTPenangMalaysia, Penang, Malaysia T&T Graduate Analyst - Artificial Intelligence & Data - MY - Sep 2025.Service Line / Portfolios : Engineering, AI & Data.
Are you ready to unleash your potential?.At Deloitte, our purpose is to make ...Tunjukkan lagiKemas kini terakhir: 2 hari yang lalu