Evaluation Scenario Writer - AI Agent Testing SpecialistMindrift • Seberang Perai, Penang, Malaysia
Evaluation Scenario Writer - AI Agent Testing Specialist
Mindrift • Seberang Perai, Penang, Malaysia
7 hari lalu
Penerangan pekerjaanBachelor’s and / or Master’s degree in Computer Science, Software Engineering, Data Science / Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / NLP, Information Systems or related fields Background in QA, software testing, data analysis, or NLP annotation Good understanding of test design principles (e.g., reproducibility, coverage, edge cases) Strong written communication skills in English Comfortable with structured formats like JSON / YAML for scenario description Can define expected agent behaviors (gold paths) and scoring logic Basic experience with Python and JavaScript Curious and open to working with AI‑generated content, agent logs, and prompt‑based behavior Ready to learn new methods, able to switch between tasks and topics quickly, and sometimes work with challenging, complex guidelines Fully remote freelance role – only requires a laptop, internet connection, available time, and enthusiasm to take on a challenge Experience in writing manual or automated test cases Familiarity with LLM capabilities and typical failure modes Understanding of scoring metrics (precision, recall, coverage, reward functions) Get paid for your expertise, with rates up to $38 / hr depending on your skills, experience, and project needs Participate in a flexible, remote, freelance project that fits around your primary professional or academic commitments Gain valuable experience to enhance your portfolio through an advanced AI project Influence how future AI models understand and communicate in your field of expertise
Mindrift is looking for a freelance Agent Scenarios Designer based in the specified country. The role focuses on designing realistic and structured evaluation scenarios for LLM‑based agents, testing agent outputs, and refining tests. You will work on a flexible schedule and receive pay up to $38 / hr based on experience.
What We Do
The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting‑edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real‑world expertise from across the globe.
About the Role
You will design realistic and structured evaluation scenarios, create test cases that simulate human‑performed tasks, and define gold‑standard behavior to compare agent actions against. Your work will ensure each scenario is clearly defined, well‑scored, and easy to execute and reuse. You need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.
Responsibilities
- Design structured test scenarios based on real‑world tasks
- Define the golden path and acceptable agent behavior
- Annotate task steps, expected outputs, and edge cases
- Work with developers to test scenarios and improve clarity
- Review agent outputs and adapt tests accordingly
How to Get Started
Apply to this posting, qualify, and you’ll have the chance to contribute to projects aligned with your skills on your own schedule. From creating training prompts to refining model responses, you’ll help shape the future of AI while ensuring technology benefits everyone.
Requirements
Nice to Have
Benefits
#J-18808-Ljbffr
Buat amaran kerja untuk carian ini
Evaluation Writer Ai • Seberang Perai, Penang, Malaysia
Pekerjaan berkaitan%20SDN%20BHD)
%20Sdn%20Bhd)
Content Creator
DK UKTIMATE (M) SDN BHD • Sungai Petani, Kedah, Malaysia
Diploma in Marketing, Communications, or a related field (or equivalent experience).Proven experience creating content across different platforms (social media, blogs, websites, video, etc.Strong w...Tunjukkan lagi
Kemas kini terakhir: 16 hari yang lalu • Dinaikkan pangkat
Senior Full Stack Engineer (AI-Native) - Contract Role
Mindvalley, Inc. • Central George Town, Penang, Malaysia
We’re looking for a Senior Full Stack Engineer who is highly trained in AI — someone who can.This is not a typical engineering role.
You will operate like a technical co‑founder, working directly wi...Tunjukkan lagi
Kemas kini terakhir: 1 jam yang lalu • Dinaikkan pangkat • Baharu!
AI-Powered Full-Stack Engineer — Prototyping to Production
Mindvalley, Inc. • Sungai Petani, Kedah, Malaysia
A leading innovative technology company based in Malaysia is seeking a Senior Full Stack Engineer focused on AI product development.
You will work closely with the Innovation Team to create impactfu...Tunjukkan lagi
Kemas kini terakhir: 1 jam yang lalu • Dinaikkan pangkat • Baharu!
Azure Architect (AI Adoption / Security)
Softenger (Malaysia) Sdn Bhd • Kulim, Kedah, Malaysia
Job Title : AI Architect (Adoption / Security).We are hiring for key roles to support a major enterprise‑scale AI transformation program.
Candidates will work closely with business and IT teams to driv...Tunjukkan lagi
Kemas kini terakhir: 3 hari yang lalu • Dinaikkan pangkat
Senior Specialist / Specialist - Test (Impedance Measurement)
AT&S • Kulim, Kedah, Malaysia
Senior Specialist / Specialist - Test (Impedance Measurement).Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.
IC substrates and printed circuit boards....Tunjukkan lagi
Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat
Become a Luxury Brand Evaluator Automobile Project in Penang, Malaysia
CXG • George Town, Penang, Malaysia
Are you a luxury automobile enthusiast who appreciates the finer details of high-end vehicles? If the answer is yes, we are looking for you! As a Luxury Brand Evaluator, you will step into the worl...Tunjukkan lagi
Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat
Hybrid ML Engineer - Diffusion & Vision (Remote-Friendly)
Bjak • Batu Kawan, Penang, Malaysia
A leading AI company in Malaysia seeks a Machine Learning Engineer to develop cutting-edge generative vision features.You'll customize diffusion models and build large-scale datasets while collabor...Tunjukkan lagi
Kemas kini terakhir: 1 jam yang lalu • Dinaikkan pangkat • Baharu!
Nahuatl Language Expert - AI Trainer
Invisible Expert Marketplace • Bayan Lepas, Penang, Malaysia
Nahuatl Language Expert – AI Trainer.We’re looking for Nahuatl language specialists who live and breathe regional dialects, phonetics, syntax, idiomatic expressions, oral traditions, and cultural r...Tunjukkan lagi
Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat
AI-Powered Market Outreach SDR (Hybrid)
S&P Global • George Town, Penang, Malaysia
A leading financial information provider in George Town is looking for an Associate in Market Outreach to drive sales development initiatives.
This role involves managing inbound inquiries, making c...Tunjukkan lagi
Kemas kini terakhir: 1 hari yang lalu • Dinaikkan pangkat
Remote SEO Content Writer for Growth and Impact
Alphaiotabpo • Sungai Petani, Kedah, Malaysia
A dynamic digital marketing firm based in Kuala Lumpur is seeking a content creator with expertise in social media advertising.
The role involves researching, writing, and optimizing content while a...Tunjukkan lagi
Kemas kini terakhir: 4 hari yang lalu • Dinaikkan pangkat
Senior Research Engineer Multimodal & Video Foundation Model (100% Remote)
Tether Operations Limited • Malacca, 04, MY
Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution.
Our cutting-edge solutions empower businesses—from ex...Tunjukkan lagi
Kemas kini terakhir: 30+ hari yang lalu
Credit Evaluation & Approval Specialist
AFFIN Group • PenangMalaysia, Penang, Malaysia
A leading financial institution in Malaysia, Penang, is seeking an individual to join their team for credit evaluation and document verification.
The ideal candidate will assess credit proposals, en...Tunjukkan lagi
Kemas kini terakhir: 1 hari yang lalu • Dinaikkan pangkat
Freelance Medical Content Writer
Digital • Seberang Perai, Penang, Malaysia
Hire Digital is seeking a remote.Freelance Medical Content Writer.The ideal candidate has a solid background in medical or pharmaceutical writing and can translate complex topics into clear and eng...Tunjukkan lagi
Kemas kini terakhir: 7 hari yang lalu • Dinaikkan pangkat
Evaluation Scenario Writer - AI Agent Testing Specialist
Mindrift • Central George Town, Penang, Malaysia
Mindrift is looking for a freelance.The role focuses on designing realistic and structured evaluation scenarios for LLM‑based agents, testing agent outputs, and refining tests.You will work on a fl...Tunjukkan lagi
Kemas kini terakhir: 7 hari yang lalu • Dinaikkan pangkat
Country Manager, Malaysia & Indonesia — AI & Enterprise Sales
Proto • Sungai Petani, Kedah, Malaysia
A leading AI solutions provider is seeking a Country Manager to oversee operations in Malaysia and Indonesia.The role involves engaging with prospects, managing a sales pipeline, and leading presen...Tunjukkan lagi
Kemas kini terakhir: 1 jam yang lalu • Dinaikkan pangkat • Baharu!
Content Creator Specialist - AI Trainer
Invisible Expert Marketplace • Batu Kawan, Penang, Malaysia
Content Creator Specialist - AI Trainer.Be among the first 25 applicants.Are you a creative content creator eager to shape the future of AI?.
Large-scale language and vision models are transforming ...Tunjukkan lagi
Kemas kini terakhir: 11 hari yang lalu • Dinaikkan pangkat
AI Evaluator - Cantonese (Chinese) - Malaysia
CrowdGen by Appen • Seberang Perai, Penang, Malaysia
Join CrowdGen as we launch an exciting new AI Voice Interaction Project designed to help improve the way voice assistants understand and respond to users! We’re looking for detail-oriented contribu...Tunjukkan lagi
Kemas kini terakhir: 7 hari yang lalu • Dinaikkan pangkat
Associate Research Analyst
Informa TechTarget • George Town, Penang, Malaysia
Do you want to develop your career and make an impact in the fast-growth, fast-moving B2B technology space?.At Informa TechTarget, you’ll collaborate and grow alongside some of the industry’s most ...Tunjukkan lagi
Kemas kini terakhir: 30+ hari yang lalu • Dinaikkan pangkat