Databricks+Sql

Role Description

Core Responsibilities

Data Pipeline Development Design and implement robust ETLELT pipelines using PySpark Spark SQL and Python

Performance Optimization Optimize Databricks clusters and code for efficient largescale data processing

Data Lakehouse Management Implement Delta Lake architecture for highquality data storage management and performance

Data Modeling Architecture Create scalable data models and analytical structures

System Integration Automation Integrate Databricks with data lakes eg ADLS and orchestrate workflows eg Azure Data Factory Airflow

Monitoring Support Monitor job performance troubleshoot failures and ensure data quality

Key Qualifications Skills

Experience 35 years in data engineering or related roles

Technical Proficiency Strong skills in PySpark Python SQL and Spark optimization

Platforms Handson experience with Azure Databricks preferred AWS Databricks or GCP Databricks

Tools Technologies Knowledge of Delta Lake Unity Catalog Databricks SQL and CICD tools Git

Databases Experience with Data Warehousing and Big Data technologies