Data Engineer-Databricks, Python)

Role &

Responsibilities

Key Responsibilities

• Architect and implement enterprise-grade Lakehouse solutions using Databricks
• Design and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL)
• Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworks
• Implement and optimize Databricks components: Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and Workflows
• Design large-scale data warehousing solutions with 3NF and dimensional modeling
• Establish data governance, security, and data quality frameworks, including Unity Catalog
• Lead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI)
• Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow)
• Drive CI/CD, DevOps practices, and performance optimization of Spark workloads
• Provide technical leadership, mentor teams, and ensure successful delivery
• Collaborate with stakeholders to translate business requirements into scalable solutions

Ideal Candidate

• Strong Databricks Architect Profile with end-to-end Lakehouse ownership
• Mandatory (Experience 1) – Must have 10+ years of software engineering experience with atleast 5+ years in Data Engineering with hands on exposure to Databricks and strong ownership of end-to-end data pipeline development.
• Mandatory (Experience 2) – Must have atleast 5+ years of expertise across the Databricks ecosystem — Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, Workflows, Unity Catalog
• Mandatory (Tech skill 1) – Must have worked at architecture level, owning end-to-end design through deployment
• Mandatory (Tech skill 2) – Must have strong experience with Python and SQL for data processing and Apache Spark for performance tuning & scalability
• Mandatory (Tech skill 3) – Must have experience in large-scale data warehousing & advanced data modeling (3NF and dimensional) across batch and real-time systems
• Mandatory (AI Exposure) – Must have at least a basic working understanding of how AI services or tools work
• Mandatory (Communication) – Must have strong stakeholder management & requirement-gathering experience with US or UK clients
• Mandatory (Company) – Must come from a B2B IT services or IT consulting background
• Mandatory (Note) – CTC is inclusive of 5% variable
• Preferred (Tech skill 1) – Azure Databricks or Azure data services experience (project runs on Azure DevOps)
• Preferred (Tech skill 2) – MLflow or MLOps practices and AI use cases (RAG, AI/BI)
• Preferred (Tech skill 3) – CI/CD, Databricks Asset Bundles (DABs) or equivalent packaging, Terraform or IaC, reusable deployment templates
• Preferred (Integrations) – ServiceNow or enterprise system integrations
• Preferred (Certifications) – Databricks (Data Engineer Associate or Professional, ML or GenAI tracks), Azure or AWS cloud certifications

Perks, Benefits and Work Culture

• The company provides free AWS and Azure certification training
• Group medical insurance of ₹5 lakhs is included in the benefits package, along with meal allowances

Skills: spark,data engineer,ci,enterprise,apache spark,aws,cd,azure