Senior Fullstack Data & AI Search Engineer_100% Remote

Job Title - Senior Fullstack Data & AI Search Engineer

Experience Required - 8+ Years

Timezone - Approx 1:00 PM / 2:00 PM and 10:00 PM / 11:00 PM IST (CET Time Zone)

Work Mode - Remote

Profile: Senior Data & AI Search Engineer with hands-on expertise in RAG pipelines and agentic AI workflows

Primary Focus: Elasticsearch + RAG + agentic AI workflows

Experience: Senior (8+ years total; 6+ years in enterprise search / RAG / LLM applications)

Role Overview

We are looking for a hands-on Data & AI Search Engineer to design and deliver a production-grade, AI-augmented enterprise search capability for a large international organisation. The engagement covers the full pipeline from raw data ingestion through to AI-generated, grounded answers surfaced via a conversational or search interface.

The right candidate combines deep Elasticsearch engineering with practical experience building Retrieval-Augmented

Generation (RAG) pipelines and agentic AI workflows. This is an individual contributor role with direct impact on a critical knowledge management platform.

Key Responsibilities

• Data Engineering and Ingestion
• Design and build scalable ingestion pipelines and connectors from enterprise sources including SharePoint, Liferay, web crawls, Data Lakes, and corporate systems into Elasticsearch or equivalent search indexes.
• Support batch, incremental, and near-real-time indexing; implement change tracking, version management, source provenance, access permission mapping, and deletion event handling to keep the index accurate.
• Build document conversion pipelines for PDF, Word, Excel, PowerPoint, HTML, email, and scanned content; convert to structured Markdown and vector embeddings using tools such as Marker, Docling, or equivalent frameworks.
• Design semantic chunking strategies (chunk size, overlap, section-aware splitting, heading preservation, table handling) and implement metadata extraction, enrichment, and deduplication during ingestion.
• Retrieval and Search
• Develop hybrid search capabilities combining BM25 keyword search, semantic vector search, metadata filtering, and contextual retrieval.
• Build re-ranking pipelines using embedding models, cross-encoders, or custom ranking logic to improve result relevance.
• Implement advanced retrieval techniques: query rewriting, query expansion, multi-query retrieval, parent-child retrieval, contextual document embeddings, and contextual compression.
• Enforce security controls so users retrieve only content they are authorised to access.
• RAG Pipeline and Agentic Workflows
• Design and build the end-to-end RAG pipeline connecting enterprise search to large language models for grounded answer generation.
• Implement agentic workflows where the AI can invoke tools, call enterprise APIs, perform multi-step reasoning, and refine searches iteratively to answer complex queries.
• Engineer prompt orchestration patterns: system prompts, retrieval prompts, guardrails, context assembly, response formatting, and fallback strategies for low-confidence or ambiguous queries.

Technical Requirements

Core Search Engineering

• Deep, hands-on Elasticsearch experience: query DSL, BM25 tuning, function_score, boosting and decay functions, multi-field matching.
• Index and data modelling: field type selection, custom analyzers and tokenizers per content type (code, prose, structured records, multimedia).
• Cluster operations: shard strategy, index sizing, reindexing, query latency tuning, and cluster health management.
• Search evaluation and relevance testing: building ground-truth benchmarks, measuring precision/recall, NDCG, and iterating against them.
• Experience with Elasticsearch, OpenSearch, Azure AI Search, or equivalent enterprise search platforms.

Data and Ingestion Engineering

• Proven experience building or configuring connectors for SharePoint, Liferay, databases, and Azure Data Lake including incremental sync, CDC, rate limiting, and API edge-case handling.
• Proficiency in Python; experience with data processing frameworks and document conversion libraries.

AI and RAG Engineering

• Hands-on experience with embedding models, re-ranking models, cross-encoders, prompt engineering, and response grounding techniques.
• Experience with LLM orchestration frameworks: LangChain, LlamaIndex, Haystack, or equivalent.
• Practical experience with tool calling, agentic workflows, function calling, and multi-step retrieval.
• Experience integrating with commercial or open-source LLMs: Azure OpenAI, OpenAI, Anthropic, Google Gemini, Meta Llama, Mistral, or similar.

Frontend

• Working knowledge of React or equivalent front-end technologies to support search UI integration (desirable, not mandatory).

Qualifications and Experience

• First-level university degree in Computer Science, Computer Engineering, Information Systems, or a related discipline.
• 8 years of professional experience in software or data engineering.
• Minimum 6 years of hands-on experience building enterprise search, AI-powered search, semantic search, RAG, or LLM-based applications.
• Excellent written and verbal communication skills in English.

What This Engagement Offers

• We have built a high-visibility knowledge management platform for a large international organisation.
• End-to-end ownership across data engineering, retrieval, and GenAI layers.
• Fully remote, flexible working arrangement within agreed time zone coverage.
• Potential for contract extension based on delivery and business need.