CloudAngles - Data Engineer - Azure Data Factory

Role Overview

As a Data Engineer at CLOUD ANGLES DIGITAL TRANSFORMATION INDIA PRIVATE, you will be instrumental in designing, building, and maintaining our data infrastructure. You will collaborate closely with data scientists, analysts, and other engineers to ensure the efficient and reliable flow of data across our systems. Your work will directly impact our ability to deliver valuable insights to our clients, enabling them to make informed decisions and achieve their business objectives.

Key Responsibilities

• Design and implement scalable and robust data pipelines using Azure Data Factory to ingest, transform, and load data from various sources into our data lake.
• Develop and maintain data models and schemas following the Medallion Architecture to ensure data quality and consistency.
• Build and optimize PySpark jobs for large-scale data processing and analysis to support data science initiatives.
• Collaborate with data scientists and analysts to understand their data requirements and provide them with the necessary data infrastructure and tools.
• Implement and maintain CI/CD pipelines using Jenkins and GitHub to automate the deployment of data engineering solutions.
• Monitor and troubleshoot data pipelines to ensure data quality and availability for downstream applications.
• Contribute to the development of data engineering best practices and standards to improve the efficiency and reliability of our data infrastructure.

Required Skillset

• Demonstrated ability to design, develop, and maintain data pipelines using Python and PySpark.
• Proven experience in building and deploying data solutions using Azure Data Factory.
• Expertise in implementing data models and schemas following the Medallion Architecture.
• Proficiency in using GitHub for version control and collaboration.
• Experience with CI/CD pipelines using Jenkins for automated deployments.
• Strong understanding of data warehousing concepts and data modeling techniques.
• Excellent communication and collaboration skills to work effectively with cross-functional teams.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Ability to work effectively in a hybrid work environment based in Hyderabad.