Big Data Engineer - Python/PySpark

Notice Period : Immediate - 30 days

Mandatory Skills : Big Data, Python, SQL, Spark/Pyspark, AWS Cloud

JD and required Skills & Responsibilities

• - Actively participate in all phases of the software development lifecycle, including requirements gathering, functional and technical design, development, testing, roll-out, and support.
• - Solve complex business problems by utilizing a disciplined development methodology.
• - Produce scalable, flexible, efficient, and supportable solutions using appropriate technologies.
• - Analyse the source and target system data. Map the transformation that meets the requirements.
• - Interact with the client and onsite coordinators during different phases of a project.
• - Design and implement product features in collaboration with business and Technology stakeholders.
• - Anticipate, identify, and solve issues concerning data management to improve data quality.
• - Clean, prepare, and optimize data at scale for ingestion and consumption.
• - Support the implementation of new data management projects and re-structure the current data architecture.
• - Implement automated workflows and routines using workflow scheduling tools.
• - Understand and use continuous integration, test-driven development, and production deployment frameworks.
• - Participate in design, code, test plans, and dataset implementation performed by other data engineers in support of maintaining data engineering standards.
• - Analyze and profile data for the purpose of designing scalable solutions.
• - Troubleshoot straightforward data issues and perform root cause analysis to proactively resolve product issues.

Required Skills

• - 5+ years of relevant experience developing Data and analytic solutions.
• - Experience building data lake solutions leveraging one or more of the following AWS, EMR, S3, Hive & PySpark
• - Experience with relational SQL.
• - Experience with scripting languages such as Python.
• - Experience with source control tools such as GitHub and related dev process.
• - Experience with workflow scheduling tools such as Airflow.
• - In-depth knowledge of AWS Cloud (S3, EMR, Databricks)
• - Has a passion for data solutions.
• - Has a strong problem-solving and analytical mindset
• - Working experience in the design, Development, and test of data pipelines.
• - Experience working with Agile Teams.
• - Able to influence and communicate effectively, both verbally and in writing, with team members and business stakeholders
• - Able to quickly pick up new programming languages, technologies, and frameworks.
• - Bachelor's degree in computer science