Home/Job List/AI Evaluation Engineer
Umanist NA

AI Evaluation Engineer

Umanist NA

Pune, Maharashtra, India
Full-Time
Posted 5 days ago

Job Description & Responsibilities

# AI Evaluation Engineer

  • Location:** Pune, Maharashtra
  • Work Mode:** Work From Office (WFO)(890)
  • Experience:** 9–12 Years
  • Notice Period:** Immediate to 45 Days
  • Preferred Candidates:** Local candidates based in Pune

## Role Overview

We are seeking an experienced AI Solutions Engineer to develop, evaluate, test, monitor, and support AI agents and LLM-powered applications across their development and production lifecycle.

The ideal candidate must have strong hands-on expertise in Python, LangChain, LangGraph, OpenAI, and Azure Databricks. The role involves agent orchestration, LLM evaluation, experiment lifecycle management, automated testing, observability, production monitoring, troubleshooting, and operational reliability.

This is a client-facing position involving regular interaction with clients and technical teams. Excellent verbal and written communication skills are mandatory, with no compromise on communication quality.

## Key Responsibilities

### 1. Agent Development & Orchestration

  • Design, develop, integrate, and maintain AI agents using LangChain, LangGraph, and OpenAI-based technologies.
  • Build Python-based services, REST APIs, and agent integration components using FastAPI and Pydantic.
  • Work with agent orchestration frameworks such as OpenAI Agents SDK, Google ADK, and Pydantic-AI, where applicable.
  • Support agent workflow development, integration, testing, and production operations.

### 2. LLM Evaluation & Experiment Lifecycle

  • Design and implement evaluation frameworks to assess agent quality, response accuracy, reliability, consistency, and performance.
  • Develop automated evaluation suites using pytest and custom pytest-based evaluations.
  • Work with MLflow Evaluation, DeepEval, Ragas, and promptfoo for LLM and agent evaluation.
  • Manage experiment execution, evaluation results, test iterations, and quality improvement cycles.
  • Identify performance gaps and support continuous improvement of AI agent outputs.

### 3. Agent Observability & Tracing

  • Implement tracing, monitoring, and observability for agent workflows and LLM applications.
  • Use OpenTelemetry and OpenInference for telemetry and instrumentation.
  • Work with Langfuse and LangSmith to trace agent execution, investigate failures, and analyze application behavior.
  • Monitor agent performance and troubleshoot issues across application and integration layers.

### 4. Azure AI Stack

  • Work with Azure OpenAI and Microsoft Foundry for AI application and agent-related workloads.
  • Use Azure Monitor and Application Insights for telemetry, application monitoring, diagnostics, and troubleshooting.
  • Support operational monitoring and performance analysis of Azure-hosted AI solutions.

### 5. Azure Databricks & Data Operations

  • Work with Azure Databricks for data processing, SQL-based analysis, and operational workflows.
  • Use Unity Catalog for data governance and access management.
  • Work with Databricks Workflows/Jobs and SQL Warehouses.
  • Support data-related workflows required for AI agent evaluation and operations.

### 6. CI/CD, Testing & Version Control

  • Maintain source code and version control using Git and GitLab.
  • Integrate automated tests and evaluation suites into CI/CD pipelines.
  • Support repeatable testing, deployment validation, and release quality checks.
  • Troubleshoot build, test, and deployment issues.

### 7. Production Monitoring & Reliability

  • Monitor applications and services using Grafana, Prometheus, Azure Monitor, Kibana, and alerting platforms.
  • Investigate production incidents, analyze logs and telemetry, and perform root-cause analysis.
  • Support incident response, issue resolution, performance monitoring, and operational improvements.
  • Help improve system reliability, observability, and troubleshooting efficiency.

### 8. Cloud Infrastructure, Containers & Security

  • Work with Docker, Kubernetes, and Azure Kubernetes Service (AKS) for containerized workloads.
  • Understand infrastructure-as-code practices using Terraform and Bicep.
  • Apply security and identity management practices using Microsoft Entra ID, managed identities, OAuth2, Azure Key Vault, and role-based access control (RBAC).
  • Support secure application access, credential management, and appropriate authorization controls.

### 9. Client & Stakeholder Collaboration

  • Collaborate regularly with clients, engineering teams, and other stakeholders.
  • Communicate technical issues, evaluation outcomes, incident updates, and resolutions clearly.
  • Participate in technical discussions and troubleshooting sessions.
  • Maintain professional communication and timely coordination across teams.

## Mandatory Skills / Must-Haves

The Following Skills Are Explicitly Mandatory For This Position

  • Python
  • LangChain
  • LangGraph
  • OpenAI
  • Azure Databricks

Candidates must also demonstrate excellent communication, analytical thinking, problem-solving, and troubleshooting capabilities appropriate for a client-facing engineering role.

## Nice-to-Have Skills

### Programming & APIs

  • FastAPI
  • REST APIs
  • Pydantic
  • pytest

### Agent Orchestration

  • OpenAI Agents SDK
  • Google ADK
  • Pydantic-AI

### Agent Observability

  • OpenTelemetry
  • OpenInference
  • Langfuse
  • LangSmith

### LLM Evaluation & Experiment Lifecycle

  • MLflow Evaluation
  • DeepEval
  • Ragas
  • promptfoo
  • Custom pytest evaluations

### Azure AI Stack

  • Azure OpenAI
  • Microsoft Foundry
  • Azure Monitor
  • Application Insights

### Databricks

  • Unity Catalog
  • Databricks Workflows/Jobs
  • SQL Warehouses

### CI/CD & Version Control

  • Git
  • GitLab
  • Automated testing pipelines

### Production Monitoring

  • Grafana
  • Prometheus
  • Azure Monitor
  • Kibana
  • Alerting platforms

### Cloud & Containers

  • Docker
  • Kubernetes
  • Azure Kubernetes Service (AKS)
  • Terraform
  • Bicep

### Security & Identity

  • Microsoft Entra ID
  • Managed identities
  • OAuth2
  • Key Vault
  • Role-Based Access Control (RBAC)

## Candidate Screening Criteria

  • **Experience:** 9–12 years(relevnat should be 8+yrs).
  • **Employment stability:** Minimum 2 years in an organization.
  • **Notice period:** Immediate to 45 days.
  • **Communication:** Excellent verbal and written communication is mandatory; no compromise.
  • **Location:** Local Pune candidates are preferred.
  • **Work arrangement:** Work From Office is mandatory.
  • **Shift flexibility:** willingness to work from office and in rotational shifts

Skills: langchain,azure,sql,langgraph,python,azure databricks,communication,ci/cd,openai,docker,kubernetes,rest apis,git

Required Skills

PythonAzureDockerKubernetesSQLGitCI/CDTerraformCommunication

Job Details

Employment TypeFull-Time
Work ModeOn-Site
Experience9 – 12 years
Positions1

Posted by

N/A

Posted on:

13 Nov 2026

About Umanist NA

More open roles

Browse all jobs →