AI Test Engineer - GO/JC/4683/2026

Job title: AI Test Engineer

Location: Pan India

Work Mode: Remote

Candidate Specification: 3+ years overall, Relevant – 3+ years

Candidate Location: Pan India

Skills: MCP, RAG, agentic AI, LangSmith

Candidate Industry: IT

Education and Certification: Any Graduation

Job Description

Must have 3+ years in software QA/testing, with hands-on senior-level experience evaluating LLM-based or agentic AI systems (not just traditional QA)

Direct experience with eval frameworks/tools — LangSmith strongly preferred; also acceptable: Ragas, DeepEval, PromptFoo, Braintrust, Langfuse

Strong understanding of RAG architecture and how to independently evaluate retrieval quality vs. generation quality

Practical familiarity with MCP (Model Context Protocol) or comparable agent tool calling frameworks

Strong Python skills for building test harnesses, scoring pipelines, and automation

Experience designing evaluation approaches for non-deterministic systems (probabilistic scoring, not binary pass/fail)

Comfortable working directly with client engineering teams — strong communication, able to explain quality risk to both technical and business stakeholders