Airflow Orchestration and Ingestion Engineer (India)

We are looking for an experienced Airflow Orchestration and Ingestion Engineer to support the migration of legacy data workflows from Apache Oozie to Apache Airflow, as part of a broader transition from an on-prem Hadoop setting to a modern cloud-based data platform on Databricks and AWS . This is a critical role focused on reengineering data pipeline orchestration, automation, and deployment within a cloud-native framework.

Key Responsibilities

Workflow Migration

• Convert Oozie workflows to Airflow DAGs using Python.
• Build reusable, modular Airflow pipelines for ingestion, transformation, and scheduling.
• Ensure accurate migration and one-to-one workflow alignment without disrupting business processes.

Cloud Data Platform Transition

• Work with engineering teams to migrate Hadoop workloads to Databricks on AWS.
• Leverage Airflow to orchestrate data pipelines across AWS services (S3, EMR, Glue, Redshift).

Pipeline Optimization

• Enhance pipeline performance for throughput and latency in the AWS ecosystem.
• Integrate Airflow with Databricks for transformation and analytics tasks.

Monitoring & Error Handling

• Implement retry logic, exception handling, and alerting in Airflow.
• Set up observability tools like CloudWatch, Prometheus, or Airflow s native monitoring.

Collaboration & Documentation

• Collaborate with data architects, DevOps, and cloud teams.
• Document orchestration logic, best practices, and the migration process.

CI/CD and Infrastructure Automation

• Develop CI/CD pipelines using Jenkins and Terraform.
• Automate DAG deployment and infrastructure provisioning via IaC.
• Integrate validation steps in deployment workflows.

Required Skills and Experience

• Strong expertise in Apache Airflow , including complex DAG design and orchestration.
• Prior experience with Apache Oozie and workflow migration.
• Proficiency in Python for Airflow DAG development.
• Hands-on experience with Hadoop ecosystems (e.g., HDFS, Hive, Spark).
• Knowledge of CI/CD tools such as Jenkins and Infrastructure as Code (IaC) with Terraform.
• Experience with Databricks (preferably on AWS) and big data orchestration.
• Solid understanding of AWS services : S3, EMR, Glue, Lambda, Redshift, IAM.
• Familiarity with container tools such as Docker or Kubernetes .

Preferred Qualifications

• Experience with large-scale cloud migrations , especially Hadoop-to-Databricks.
• Proficiency in Spark / PySpark for big data transformation.
• AWS or Databricks certifications are a plus.
• Familiarity with Git and workflow monitoring platforms. .