Python Pyspark Developer

About The Opportunity

A fast-scaling technology services firm operating in the Data Engineering and Cloud Analytics space, we partner with enterprises to build scalable data pipelines, automate ETL workflows, and unlock real-time insights from structured and unstructured datasets. Our Python PySpark developers architect and deploy high-performance data solutions on cloud platforms—enabling faster decision-making, predictive modeling, and regulatory compliance for global clients.

Role & Responsibilities

• Design, develop, and optimize PySpark ETL pipelines for large-scale batch and streaming data processing.
• Collaborate with data scientists and analysts to ingest, transform, and model data for ML training and reporting use cases.
• Implement data quality checks, error handling, and logging frameworks within Spark jobs for production-grade reliability.
• Integrate PySpark workflows with cloud data lakes (AWS S3, Azure Data Lake), metastores (Hive, Glue), and orchestration tools (Airflow, Luigi).
• Performance-tune Spark applications—partitioning, caching, broadcasting, and resource allocation—to reduce job runtimes and costs.
• Write clean, modular, testable Python code with unit/integration tests and document architecture decisions for team scalability.

Skills & Qualifications

Must-Have

• PySpark
• Python
• Apache Spark
• ETL Development
• SQL
• Data Lake Architecture
• AWS S3
• Apache Airflow

Preferred

• Azure Data Lake
• Databricks
• Delta Lake

Benefits & Culture Highlights

• On-site collaborative workspace in major Indian tech hubs with modern infrastructure and R&D labs.
• Opportunities to upskill in cloud-native data platforms and GenAI-powered data pipelines.
• Fast-track career growth with cross-functional exposure to data science, ML engineering, and cloud architecture teams.

Skills: data,python,lake,spark,cloud