Senior Data Engineer (Azure Databricks & PySpark, Python)

Location: Bangalore / Bengaluru, India (Hybrid)

Role: Full-Time

Experience Level: Mid–Senior Level (3–7 Years)

About the Role

We are seeking an experienced and passionate Data Engineer specializing in Azure Databricks, PySpark, and Python to join our data engineering team in Bangalore. In this role, you will design, build, and optimize scalable data pipelines, leverage modern lakehouse architectures, and transform raw data into actionable enterprise insights.

Key Responsibilities

• Pipeline Development: Design, construct, test, and maintain robust batch and real-time ETL/ELT pipelines using PySpark and Azure Databricks.
• Architecture & Optimization: Build and manage data lakehouse platforms leveraging Delta Lake; optimize Spark jobs for performance, memory, runtime efficiency, and cost.
• Azure Integration: Seamlessly integrate Databricks with Azure ecosystem components including Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), Azure Synapse, and Azure Key Vault.
• Data Quality & Governance: Enforce data quality standards, monitoring, and security models using Databricks Unity Catalog and enterprise access controls.
• DevOps & Automation: Build CI/CD deployment pipelines using Azure DevOps or GitHub Actions for continuous integration and delivery of data workloads.
• Collaboration: Partner with data architects, data analysts, and business stakeholders to translate functional requirements into scalable technical solutions.

Required Qualifications & Skills

• Core Tech Stack: Proven expertise in Python, PySpark, and Azure Databricks.
• Data Engineering Experience: Strong hands-on experience building end-to-end data pipelines and implementing Delta Lake architectures.
• Orchestration & Storage: Proficiency with Azure Data Factory (ADF) and ADLS Gen2.
• Database & Querying: Strong proficiency in SQL and deep understanding of relational/non-relational database concepts and data modeling (Kimball, Data Vault).
• Version Control & CI/CD: Hands-on experience with Git and Azure DevOps/Jenkins for code versioning and automated deployments.
• Problem Solving: Strong analytical, debugging, and query optimization capabilities.

Good to Have (Preferred)

• Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer Professional.
• Streaming Data: Experience with Spark Structured Streaming, Azure Event Hubs, or Apache Kafka.
• Data Governance: Hands-on experience with Unity Catalog or Azure Purview.

What We Offer

• Competitive salary and performance bonuses
• Flexible hybrid work culture
• Comprehensive health and wellness benefits
• Continuous learning opportunities and certification sponsorships

How to Apply

If you are passionate about building modern data platforms at scale, drop your updated CV at communications@qubistsolutions.com or apply directly via this post!

#Hiring #JobOpening #DataEngineer #AzureDatabricks #PySpark #Python #BangaloreJobs #BengaluruJobs #DataEngineering #AzureDataEngineer #BigData #ETL #DeltaLake #CloudData #AzureJobs #TechJobsIndia #Careers #DataJobs #WorkInBangalore #HiringDataEngineers #IndiaJobs #DataArchitecture #DevOps #SoftwareEngineering