Lead Cloud Data Engineer (Hyderabad)

Job Title: Senior Data Engineer Employment Type: Full Time, Permanent Position Location: Hyderabad Qualifications: BE/B.Tech/MCA Degree in Computer Science, Engineering, or similar relevant field Total Experience: 8 to 14 years Working Model: Work from office Data Pipeline Development - Design and implement end-to-end batch and streaming data pipelines using PySpark and Apache Spark. - Build and manage data workflows on Databricks, including Delta Lake architecture and medallion (Bronze/Silver/Gold) patterns. - Develop and maintain ETL/ELT processes to ingest, transform, and load data from diverse sources. - Optimize Spark jobs for performance, cost-efficiency, and reliability at scale. Cloud & Azure Services - Manage data solutions using Azure services including Azure Data Lake Storage (ADLS Gen2), Azure Event Hubs, and Azure Key Vault. - Ensure data security, access control, and compliance within Azure environments. - Monitor and troubleshoot cloud infrastructure, pipelines, and data flows. Data Modeling & SQL - Design and implement dimensional and normalized data models for analytical and operational use cases. - Write complex SQL queries for data transformation, validation, and reporting. - Work with structured, semi-structured, and unstructured data across various storage formats (Parquet, Delta, JSON, CSV). Required Skill - 6+ years of hands-on experience in data engineering roles. - Strong expertise in Apache Spark and PySpark for large-scale data processing. - Deep experience with Databricks platform — workspace setup, clusters, jobs, Delta Live Tables. - Solid experience in building Azure Data Factory Pipeline. - Proficiency with Azure cloud services for data engineering workloads. - Advanced SQL skills including window functions, CTEs, and query optimization. - Solid Python programming skills for data engineering, scripting, and automation. - Experience with version control (Git) and CI/CD pipelines for data code deployments. - Familiarity with d