Data Engineer Scala | Kafka | Spark | GCP (Bengaluru)

We are looking for an experienced Data Engineer with 6+ years of experience in designing, developing, deploying, monitoring, and optimizing scalable batch and streaming data pipelines. The ideal candidate should have strong hands-on expertise in Scala, Apache Spark, Kafka, Airflow, BigQuery, Dataproc, and GCP.

Key Responsibilities

• Design, develop, deploy, monitor, and optimize ETL/ELT workflows for both batch and real-time/streaming data processing.
• Develop scalable data processing solutions using Scala and Apache Spark.
• Build and manage streaming pipelines using Kafka, including handling high-volume data and back-pressure scenarios.
• Develop and maintain Airflow DAGs for complex data orchestration, dependency management, scheduling, retries, and idempotent execution.
• Work extensively with Google Cloud Platform (GCP) services including BigQuery and Dataproc.
• Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality validations, transformations, aggregations, and summarization.
• Perform performance tuning and optimization of Spark jobs, Kafka pipelines, BigQuery queries, and ETL workflows.
• Implement monitoring and observability mechanisms to identify and troubleshoot pipeline failures, performance issues, and data-quality problems.
• Create and maintain runbooks and operational documentation for production support and continuous optimization.
• Troubleshoot runtime data issues independently and ensure timely resolution of production incidents.
• Follow best practices for data governance, security, access control, and information security.

Required Skills

• 6+ years of experience as a Data Engineer.
• Strong hands-on experience with Scala and Apache Spark.
• Solid knowledge of Kafka and streaming data processing.
• Experience with Apache Airflow and workflow orchestration.
• Hands-on experience with Google Cloud Platform (GCP).
• Strong experience with BigQuery and Dataproc.
• Experience building both batch and streaming ET .