Technical AI Evaluation Analyst (LATAM)
About Gramian
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
About the Role
We are seeking a detail-oriented AI Evaluation & Quality Assurance Specialist to review the quality, correctness, and fairness of tasks designed to evaluate AI agents. You will inspect reference solutions, grading logic, execution traces, and generated deliverables to identify task defects, evaluation errors, and unjustified model failures. This role requires strong technical fluency, independent analytical judgment, and the ability to produce clear, evidence-based feedback.
LOCATION: Remote – Latin America (LATAM)
CONTRACT: Hourly Contractor
COMMITMENT: 40 hours per week
TIME OVERLAP: 8 hours of mandatory PST overlap
DURATION: 10 weeks
START DATE: Immediately
Key Responsibilities
- Validate task quality by checking instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
- Review AI agent execution traces, tool calls, and deliverables to determine wheth
