Python Expert with AWS Data Engineering

We are seeking a Python Data Engineering expert with 8+ years of overall experience in Data Engineering and at least 8+ years of hands-on expertise in Python development. The ideal candidate should have strong proficiency in Python, RDBMS, and a solid understanding of AWS services.

Key Responsibilities

Python Expertise: Leverage 8+ years of hands-on experience with Python for developing scalable, high-performance data engineering solutions.

RDBMS to NoSQL Conversion: Lead the effort in migrating large data sets from RDBMS to NoSQL databases, ensuring seamless and efficient data storage and retrieval.

Data Lake & Delta Tables: Design and build data lakes and manage configurations on Delta tables to optimize data processing and storage workflows.

Cost Optimization: Implement computing strategies that minimize costs while maximizing data processing efficiency within cloud environments like AWS.

Holistic Framework Development: Analyze current environments and use cases to develop holistic data engineering frameworks that address both short-term and long-term business needs.

Airflow Framework: Utilize Apache Airflow to manage, schedule, and monitor workflows, ensuring data pipelines are running efficiently.

AWS Service Integration: Work extensively with AWS services such as EMR, S3, Lambda, Step Functions, and Aurora (RDS) to build, manage, and optimize data solutions on the cloud.

Required Skills

8+ Years of Data Engineering Experience: A strong background in Data Engineering, with 8+ years of Python development experience.

RDBMS Expertise: Proficiency in RDBMS and SQL for managing large-scale relational databases.

NoSQL: Familiarity with converting data from RDBMS to NoSQL databases to enable flexible, scalable data storage solutions.

Data Lakes & Delta Tables: Experience in building data lakes and working with Delta table configurations for optimized data storage and processing.

AWS Services: Hands-on experience with AWS technologies such as EMR, S3, Lambda, Step Functions, and Aurora (RDS) for cloud-based data solutions.

Apache Airflow: Strong understanding of Apache Airflow to automate, schedule, and manage data workflows.

Soft Skills

Team Collaboration: Demonstrated ability to work as part of a team, with a positive attitude and commitment to collaborative problem-solving.