Job Description
- Bachelor's degree in Computer Science, Engineering, or a related field. Master's degree preferred.
- Strong coding experience in Python, with proficiency in writing efficient and maintainable code for data processing and manipulation.
- Extensive experience working with GCP services, particularly BigQuery, Dataflow, and Airflow.
- Solid understanding of SQL and database concepts, with experience in optimizing SQL queries for performance.
- Experience designing and implementing Directed Acyclic Graphs (DAGs) for workflow orchestration and scheduling.
- Familiarity with data modeling concepts and techniques for both structured and unstructured data.
- Strong analytical and problem-solving skills, with the ability to quickly diagnose and resolve technical issues.
- Excellent communication and collaboration skills, with the ability to work effectively in a fast-paced, team-oriented environment.
Job Responsibilities
- Design, develop, and implement scalable and reliable data pipelines on GCP using tools such as BigQuery, Dataflow, and Airflow.
- Collaborate with data scientists and analysts to understand data requirements and translate them into technical solutions.
- Optimize and tune existing data pipelines for performance, scalability, and cost-effectiveness.
- Write clean, efficient, and maintainable code in Python for data processing, transformation, and integration.
- Implement and maintain data quality checks and monitoring processes to ensure data integrity and reliability.
- Create and maintain documentation for data pipelines, workflows, and infrastructure configurations.
- Troubleshoot and debug data pipeline issues in a timely manner, ensuring minimal disruption to data operations.
- Stay updated on the latest trends and best practices in data engineering and cloud technologies, and actively contribute to knowledge sharing within the team.
Location
Remote, India