Lead/ Senior Data Engineer ( Actvie Databricks required) _ Exp: 7+

Role & responsibilities

• Design, build, and implement Databricks-based data ingestion and transformation pipelines using Delta Lake, Delta Live Tables (DLT), Auto Loader, and Unity Catalog.
• Migrate legacy SQL Server ETL workloads and file-based data pipelines to a scalable Databricks Lakehouse architecture.
• Develop incremental data ingestion solutions using Delta Change Data Feed (CDF) to replace traditional file-based change detection mechanisms.
• Build and optimize bronze, silver, and gold data layers following Lakehouse best practices.
• Configure and optimize JDBC connectivity between Databricks and SQL Server for large-scale enterprise datasets.
• Design scalable partitioning, clustering (Z-Ordering), and performance optimization strategies for high-volume Delta tables.
• Develop Delta MERGE processes to support inserts, updates, and deletes while ensuring data consistency.
• Replace legacy Windows Task Scheduler jobs with automated Databricks workflows and Delta Live Tables pipelines.
• Perform data validation, reconciliation, row-count verification, and parallel-run testing to ensure migration accuracy.
• Enable secure enterprise data sharing using Unity Catalog, Databricks SQL Warehouse, and JDBC connectivity.
• Collaborate with business stakeholders, application owners, architects, and platform teams to deliver end-to-end migration solutions.
• Document architecture, migration strategies, operational runbooks, and production support procedures.