D
Posted 1w ago•Pathum Wan, Bangkok, Thailand
QA Automation Engineer (Senior)
SeniorOn-site (Pathum Wan)Salary undisclosed
Required Skills
TypeScriptPythonJavaSQLLLMs
Job Description
- Build evaluation suites for LLM and ML features: curated test sets, scoring against reference answers, LLM-as-judge where it is justified, and human review where it is not.
- Define what "good enough" means with the delivery team before launch, in numbers a client can accept, and hold releases to it.
- Test retrieval pipelines end to end, covering chunking and indexing, retrieval quality, grounding of the final answer, and behaviour when the right document is simply not there.
- Write regression tests that tolerate non-deterministic output, using semantic similarity, structured-output validation, tolerance bands, and repeated sampling rather than exact string matching.
- Cover failure modes that matter to clients: hallucination, prompt injection through user or document content, leakage of data across tenants or sessions, refusal and over-refusal, and degradation on Thai-language input.
- Track cost and latency alongside accuracy, so that a quality improvement that triples token spend is visible before it ships.
- Detect drift after release by comparing production samples against the baseline, and raise it early rather than after the client does.
- Design and maintain automated test suites for APIs, services, data pipelines, and web applications built for client projects.
- Run tests in CI, keep them fast and stable, and treat a flaky suite as a defect in its own right.
- Set the test strategy at the start of an en
Ready to apply? Optimize your CV for this specific jobAI customizes your experience bullets and increases chances to get hired.
