Contractor - Python & PySpark Developer

Location: Pan India

Experience: 5–7 Years

About The Role

We are looking for an experienced Python and PySpark Developer with strong hands-on expertise in PySpark ETL, Python, Spark SQL, DataFrames, and Delta Lake. The role focuses on developing, enhancing, testing, and supporting data processing pipelines, with an emphasis on using GenAI tools to improve development, debugging, testing, documentation, and code quality.

Job Description

The role involves maintaining and enhancing PySpark ETL jobs and troubleshooting data processing using Spark DataFrames, Spark SQL, and Delta Lake. You will work with file ingestion processes, including CSV, XML, and Hyper extracts, along with reporting-date handling and SQL-to-PySpark integration.

You will also develop and execute PyTest-based unit and integration tests and troubleshoot production issues using Python logging and exception handling. The position requires effective use of AI coding assistants such as GitHub Copilot, ChatGPT, or similar tools for code development, analysis, testing, debugging, documentation, and code reviews.

The role also emphasizes responsible AI usage, including data privacy, secure handling of production data, validation of AI-generated outputs, and awareness of prompt engineering for technical tasks.

Key Responsibilities

• Maintain and enhance PySpark ETL jobs and data processing pipelines
• Develop and troubleshoot Spark DataFrame, Spark SQL, and Delta Lake processing
• Implement file ingestion for CSV, XML, and Hyper extracts
• Work with SQL-to-PySpark integration and reporting-date handling
• Develop and execute PyTest-based unit and integration tests
• Troubleshoot production issues using Python logging and exception handling
• Use AI coding assistants to accelerate development, debugging, testing, code review, and documentation
• Analyze legacy code, perform impact assessments, and support root-cause analysis using AI-assisted techniques

Required Skills

• Python
• PySpark
• Spark DataFrames and Spark SQL
• Delta Lake
• ETL development
• Python debugging, logging, and exception handling
• SQL-to-PySpark integration
• File ingestion and processing
• Python packaging and runtime dependencies
• PyTest-based testing
• Code quality tools such as Pylint
• AI coding assistants such as GitHub Copilot or ChatGPT
• GenAI-assisted code analysis, testing, debugging, and documentation
• Prompt engineering for technical tasks
• Responsible AI practices and production data privacy

Good to Have

• Exposure to ML/AI concepts
• Embeddings
• Vector search
• RAG
• LLM-based knowledge assistants for internal documentation and search

Mandatory Skills

• Python

Candidate Profile

The ideal candidate should have 5–7 years of experience in Python and PySpark development, with strong hands-on expertise in ETL processing, Spark, Delta Lake, testing, and production troubleshooting. The candidate should also be comfortable using GenAI and AI coding assistants for development, analysis, testing, documentation, and code-quality improvement, with strong communication skills.

Benefits Package

• Competitive salary and benefits package
• Opportunities for professional growth and development
• A collaborative and inclusive work environment
• The opportunity to work on exciting and challenging projects with leading clients