Lead I - Data Engineering - Azure Data Factory, Pyspark, Azure Databricks (India)

Role description

UST is looking for a highly skilled Azure Data Engineer with 5 to 10 yrs of exp to join our team and play a key role in designing and implementing efficient data solutions on the Azure cloud platform. The ideal candidate should have a solid background in data engineering, possess expertise in Azure services, and demonstrate proficiency in data processing technologies, particularly PySpark.

Responsibilities

Data Pipeline Development

• Design, implement, and optimize end-to-end data pipelines on Azure, focusing on scalability and performance.
• Develop and maintain ETL workflows for seamless data processing.

Azure Cloud Expertise

• Utilize Azure services such as Azure Data Factory, Azure SQL Database, and Azure Databricks for effective data engineering.
• Implement and manage data storage solutions on Azure.

Data Transformation with PySpark

• Leverage PySpark for advanced data transformations, ensuring high-quality and well-structured output.
• Implement data cleansing, enrichment, and validation processes using PySpark.

Performance Optimization

• Optimize data pipelines, queries, and PySpark jobs to enhance overall performance and scalability.
• Identify and address performance bottlenecks within data processing workflows.

Requirements

• Proven experience as a Data Engineer, emphasizing expertise in Azure.
• Proficiency in Azure Data Factory and other relevant Azure services.
• Expertise in PySpark for data processing and analytics is a must.
• Experience with data modeling, ETL processes, and data warehousing.

Skills

Azure Data Factory, Pyspark, Azure Databricks

About UST

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.