Technical Lead – Data Engineer

Technical Lead – Data Platform | AWS / Azure / GCP

Location: Bangalore

Experience: 8+ Years

Employment Type: Permanent

Role: Technical Lead – Data Platform

Role Overview

We are looking for a hands-on Technical Lead – Data Platform with strong experience in Data Engineering, Big Data, Databricks, Apache Spark, and Cloud platforms (AWS/Azure/GCP).

The ideal candidate should be capable of leading data platform development, designing scalable data pipelines, mentoring engineers, and working across lakehouse, ETL/ELT, batch, streaming, and cloud data platforms.

Key Responsibilities

• Lead the design, development, and implementation of scalable data platforms.
• Own end-to-end data pipelines covering batch, streaming, CDC, APIs, files, and databases.
• Develop and optimize production-grade pipelines using Python, SQL, PySpark, and Apache Spark.
• Design and implement ETL/ELT frameworks and data ingestion solutions.
• Work with Databricks and cloud-native data platforms across AWS, Azure, and/or GCP.
• Design Data Lake/Lakehouse architectures using Medallion Architecture.
• Work with technologies such as Delta Lake / Apache Iceberg for scalable data processing.
• Lead technical discussions, solution design, estimations, code reviews, and architectural decisions.
• Mentor and provide technical guidance to Data Engineers.
• Establish coding standards, testing frameworks, CI/CD practices, and reusable engineering patterns.
• Implement data quality, validation, monitoring, governance, security, and compliance.
• Troubleshoot complex production issues and optimize Spark jobs, pipelines, infrastructure, and SQL workloads.
• Work closely with architects, product teams, business stakeholders, and engineering teams.
• Drive continuous improvements in performance, scalability, reliability, and cloud cost optimization.

Must-Have Technical Skills

• 8+ years of experience in Data Engineering / Big Data / Data Warehousing.
• Strong hands-on experience with Apache Spark / PySpark.
• Advanced Python and SQL skills.
• Strong experience with Databricks.
• Strong understanding of Data Lake / Lakehouse / Medallion Architecture.
• Hands-on experience with ETL/ELT and data pipeline development.
• Experience with one or more major cloud platforms:
• AWS
• Azure
• GCP
• Strong understanding of cloud services covering compute, storage, networking, IAM/security, and monitoring.
• Experience with Airflow or other workflow orchestration tools.
• Experience with Kafka / streaming / CDC is highly desirable.
• Knowledge of Delta Lake / Apache Iceberg.
• Experience with CI/CD, Git, testing, and deployment automation.
• Understanding of data quality, governance, security, and observability.