Data Engineer-5-8Yrs-Delhi/NCR

Responsibilities • Design, Develop and maintain ETL pipelines using Pyspark in Azure Databricks using delta tables.

• Create build from Github and release pipeline for Ingestion and Databricks using Azure Devops / Harness • Monitor Performance of ETL Jobs, resolve any issue that arose and improve the performance metrics as needed.
• Diagnose system performance issue related to data processing and implement solution to address them.
• Collaborate with other teams to ensure successful integration of data pipelines into larger system architecture requirement.
• Maintain integrity and quality across all pipelines and environments.
• Understand and follow secure coding practice to make sure code is not vulnerable In this role, you will report to the Data Engineering Team Lead.

Skills • Should have Strong background in Software development with experience in ingest, transform and store data from large datasets using Pyspark in Azure Databricks with strong knowledge on distributed computing concepts.

• Must have hands on experience in designing and developing ETL Pipelines in Pyspark in Azure Databricks with strong python scripting exposure like list comprehensions, Dictionary variables etc.
• Must have minimum 5 years of exposure and good proficiency in data warehousing concepts.
• Proficient in SQl and database Design concepts. EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.201.748.4700 www.exlservice.com • Hands on experience with good proficiency in Delta table and delta file operations like merge, Insert override, Partition overrides etc.
• Hands on Experience in CICD in Azure DevOps/Harness and ADF/Stone branch for orchestration.
• Knowledge of Azure cloud computing platform with Azure Synapse, ADLS.
• Knowledge on GitHub and build management.
• Passion for data and experience working within a data driven organization • Excellent presentation, communication (oral & written), and relationship building skills, across all levels of management