Senior Data Engineer Data Pipelines, Orchestration, System Design & A (Kolkata)

Senior Data Engineer: Data Pipelines, Orchestration, System Design & A (Kolkata)

Company: Applied Research Works India

Job Location: Kolkata, West Bengal, India

Employment Type: Full-time

Experience Required: 8 - 10 years

Senior Data Engineer: Data Pipelines, Orchestration, System Design & AI Tooling

Location: Kolkata

Experience: 8-10 years

Employment Type: Full-time

About the Role

We are looking for a Senior Data Engineer to design, build, and operate reliable data pipelines and orchestration frameworks, and to enable AI-driven products and workflows on top of them. You will own the data flow from ingestion to consumption, set engineering standards for the team, and work closely with analytics, data science, and ML teams.

Key Responsibilities

Data pipeline design and development

Design, build, and maintain scalable batch and streaming pipelines for structured and unstructured data.

Develop robust ingestion, transformation, and loading (ETL/ELT) processes from diverse sources (databases, APIs, event streams, files).

AWS services (EC2, ECS, Lambda, API Gateway, SQS etc)

Optimize pipeline performance, cost, and reliability

Data orchestration

Build and manage workflows using orchestration tools

Implement scheduling, dependency management, retries, alerting, and backfill strategies.

Establish monitoring, logging, and observability for pipeline health.

AI tools and enablement

Build data foundations for machine learning and generative AI use cases, including feature pipelines and data preparation for model training and inference.

Integrate AI/LLM tools and APIs into data workflows (e.g., document processing, enrichment, classification, summarization).

Support retrieval-based AI solutions, including embedding pipelines and vector databases

Collaborate with data scientists and ML engineers to move models from prototype to production.

Data quality, governance, and leadership

Implement data quality checks, validation, lineage, and documentation.

Apply security, privacy, and compliance best practices to data handling.

Lead design reviews and code reviews, and mentor junior and mid-level engineers.

Translate business requirements into technical solutions and communicate with stakeholders.

Required Qualifications

8-10 years of professional experience in data engineering or a closely related field.

Robust proficiency in Python and advanced SQL.

Proven experience building and maintaining production data pipelines at scale.

Hands-on experience with at least one orchestration tool

Experience with big data and processing frameworks

Experience with cloud platforms and their data services.

Solid understanding of data warehousing and lakehouse concepts, data modeling, and tools.

Working knowledge of AI/ML workflows and tools, including LLM APIs, embeddings, and vector databases.

Experience with version control, CI/CD, and infrastructure as code (e.g., Git, Terraform, Docker).

Strong problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

Experience building RAG or agent-based solutions in production.

Knowledge of data governance, cataloging, and lineage tools.

Cloud or data platform certifications.

Experience in healthcare industry.

Bachelor's or Master's degree in Computer Science, Engineering, or a related field. .