Quality Lead

Overview

Thoth AI Federal Territory of Kuala Lumpur, Malaysia

We are seeking a highly skilled and analytical Lead QA to oversee annotation quality across one of our key data domains — Audio, Video, or LLM . The Lead QA plays a crucial role in ensuring that all outputs from QA and annotators meet the highest standards of accuracy, consistency, and compliance with client rubrics. This position also serves as the key quality gatekeeper, collaborating closely with Trainers, Project Managers (PMs), and QA teams to drive continuous improvement in annotation performance.

Responsibilities

• Lead and manage a team of QAs within the assigned project domain (Audio / Video / LLM), ensuring quality objectives and SLAs are consistently met.
• Develop and standardize QA rubrics, error typologies, and review guidelines based on project-specific annotation requirements.
• Conduct calibration sessions regularly with QAs, Trainers, and PMs to maintain consistent quality interpretations across teams.
• Analyze QA reports and annotation error trends to identify systemic issues and recommend corrective actions.
• Collaborate with Trainers to translate recurring quality gaps into targeted training or retraining plans.
• Perform quality audits on both QA and annotator performance to ensure review accuracy and reliability.
• Monitor and report key quality metrics (Accuracy, Consistency, Disagreement Rate, Rejection Rate, etc.) to stakeholders.
• Design and optimize QA sampling strategies to ensure efficient yet effective quality coverage.
• Support new project launches by developing quality validation processes, test datasets, and pilot evaluation rubrics.
• Drive continuous improvement initiatives through process standardization, tool optimization, and cross-domain knowledge sharing.
• Ensure data integrity and confidentiality in line with client and internal security requirements.
• Act as the domain quality expert and key escalation point for quality-related client or internal concerns.

Qualifications

Required :

Preferred :

Core Competencies

Purpose of the Role

The Lead QA ensures that all annotation outputs meet defined quality benchmarks and client expectations. By managing QA teams and collaborating across training and project delivery, this role safeguards data accuracy and consistency—fundamental to the reliability and scalability of AI model training.

Seniority level

Employment type

Job function

Industries

#J-18808-Ljbffr