Sr. Data Engineer (Bengaluru)

Data Engineer – Data PlatformLocation: BangaloreExperience: 5+ YearsRole OverviewWe are looking for a hands-on Data Engineer to design, develop, and optimize scalable data platforms and data pipelines. The ideal candidate will have strong expertise in Python, SQL, PySpark/Spark, Databricks, and modern data engineering practices.Candidates should have hands-on experience with at least one cloud platform – AWS, Azure, or GCP. Experience with BigQuery, Lakehouse architecture, Delta Lake, Airflow, Kafka, and CDC will be highly valued.Key Responsibilities- Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, PySpark, and Apache Spark.- Build and optimize data processing solutions using Databricks and cloud-native data services.- Develop batch and real-time data ingestion pipelines using technologies such as Airflow, Kafka, CDC, and other streaming/integration frameworks.- Implement modern Lakehouse and Medallion Architecture using Delta Lake or similar technologies.- Develop reliable data pipelines for structured and semi-structured data from multiple sources.- Implement data quality, validation, reconciliation, monitoring, logging, and error-handling mechanisms.- Optimize Spark jobs and data pipelines for performance, scalability, reliability, and cloud cost efficiency.- Work with cloud storage, compute, databases, and data services across AWS, Azure, or GCP.- Implement engineering best practices including Git, CI/CD, unit testing, integration testing, and code reviews.- Troubleshoot data pipeline failures and resolve performance and data-quality issues.- Collaborate with Data Architects, Solution Architects, Analysts, Developers, and Business Stakeholders to understand requirements and deliver scalable data solutions.- Contribute to technical design, documentation, coding standards, and continuous improvement of the data platform.Must-Have Skills - 5+ years of hands-on experience in Data Engineering.- Strong programming skills in Python.- Robust expertise in SQL and data manipulation.- Hands-on experience with PySpark / Apache Spark.- Strong hands-on experience with Databricks.- Good understanding of ETL/ELT concepts, data pipelines, data modeling, and distributed data processing.- Experience with Lakehouse Architecture, Medallion Architecture, and Delta Lake.- Hands-on experience with at least one cloud platform: AWS / Azure / GCP.- Experience with one or more cloud data/storage services such as:- AWS: S3, Glue, EMR, Redshift- Azure: ADLS, Data Factory, Synapse- GCP: GCS, BigQuery, Dataflow- Exposure to Apache Airflow or other workflow orchestration tools.- Experience with Kafka, CDC, or real-time/streaming data pipelines.- Good understanding of data quality, monitoring, testing, and pipeline optimization.- Experience with Git and CI/CD pipelines.- Strong analytical, troubleshooting, and problem-solving skills.- Good communication and collaboration skills.Good-to-Have Skills - Apache Iceberg / BigLake- dbt- Databricks Unity Catalog- Kafka / Structured Streaming- Change Data Capture (CDC) tools- Infrastructure as Code (IaC) – Terraform or similar- Cloud security, IAM, networking, and access management- Cloud cost optimization- Data governance and metadata management- Experience working with data warehouses and dimensional data modelingPreferred Technical Stack