Data Engineer

Optomi in partnership with a leading professional services firm, is seeking a highly skilled Senior Data Engineer with deep expertise in AWS, Python, PySpark, and SQL. The ideal candidate will have experience building scalable ETL/ELT solutions, designing data models utilizing SCD Type 1 and Type 2 methodologies, and developing API-driven solutions using Python frameworks such as Fast API or Flask. This engineer will work closely with cross-functional teams to design, deploy, and maintain cloud-native applications and data platforms while ensuring reliability, scalability, and operational excellence.

What the Right Candidate Will Enjoy

• Building scalable cloud-native solutions within AWS.
• Working extensively with Python, PySpark, and modern data engineering technologies.
• Designing and deploying APIs that support enterprise applications.
• Leveraging AI-powered development tools such as GitHub Copilot.
• Collaborating with highly technical engineering and data teams.
• Solving complex data challenges within a modern cloud environment.

What Type of Experience Does the Right Candidate Have

• 6+ years of Data Engineering experience.
• Strong experience with AWS services including ECS, Lambda, SNS, and DynamoDB.
• Advanced Python programming skills.
• Hands-on experience with PySpark and distributed data processing.
• Strong SQL development and optimization experience.
• Deep understanding of data modeling concepts, including SCD Type 1 and Type 2.
• Experience building and deploying APIs using FastAPI, Flask, or similar Python frameworks.
• Experience implementing CI/CD processes and deployment automation.
• Experience working with AI-assisted development tools such as GitHub Copilot.
• Strong troubleshooting, analytical, and problem-solving abilities.

What the Responsibilities Are of the Right Candidate

• Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
• Investigate, troubleshoot, and optimize complex data pipelines.
• Develop and optimize SQL queries supporting enterprise reporting and analytics.
• Design and implement data models utilizing SCD Type 1 and Type 2 methodologies.
• Build, deploy, and support APIs within AWS environments.
• Develop solutions utilizing AWS services such as ECS, Lambda, SNS, and DynamoDB.
• Implement and maintain CI/CD pipelines to support automated deployments.
• Utilize AI-powered development tools to enhance development efficiency and code quality.
• Collaborate with cross-functional engineering teams to deliver scalable cloud-based solutions.
• Monitor, support, and continuously improve platform performance and reliability.