Walk-in || Data Engineer

Mandatory Skills: Azure Databricks, PySpark, Python, SQL, Azure Data Factory, Azure Data Lake Storage (ADLS), Spark

Job Description

• Design, develop, and maintain scalable data engineering solutions using Azure Databricks and PySpark.
• Develop and optimize PySpark applications for large-scale data processing and transformation.
• Build and maintain data pipelines using Azure Data Factory (ADF) and Databricks.
• Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing structured and unstructured data.
• Write complex SQL queries for data extraction, transformation, validation, and analysis.
• Develop ETL/ELT workflows using Databricks notebooks and Spark.
• Perform data cleansing, transformation, aggregation, and validation using PySpark.
• Optimize Spark jobs, notebooks, queries, and data pipelines for performance and scalability.
• Implement incremental data loads and handle large-volume datasets.
• Work with different data formats such as Parquet, JSON, CSV, and Delta Lake.
• Develop and maintain Delta Lake tables and implement concepts such as partitioning and schema evolution.
• Troubleshoot production data pipelines and resolve performance and data-quality issues.
• Collaborate with data architects, analysts, developers, and business teams to understand data requirements.
• Follow coding standards, security practices, CI/CD processes, and Agile methodologies.