Data Software Engineer – Spark, Python, Databricks (L2 / L4)

Data Software Engineer – Spark, Python, Databricks (L2 / L4)

Experience

• L2: 4–5 Years
• L4: 8–12 Years

Mode: FTE (Full-Time Employment)

Job Location: Bangalore

Work Mode: Hybrid

Notice Period

• L2: Immediate to 15 days
• L4: Immediate to 15 days

Drive Type: F2F

Drive Location: Bangalore

CTC Band

• L2: Up to 21 LPA
• L4: Up to 42 LPA

Role Overview

We are hiring Data Software Engineers with strong expertise in Apache Spark, Python, and AWS/Azure Databricks. The ideal candidates will have deep Big Data engineering experience, strong distributed systems knowledge, and the ability to work on complex end-to-end data platforms at scale.

Key Responsibilities

Big Data Engineering

• Design and build distributed data processing systems using Spark and Hadoop.
• Develop and optimize Spark applications, ensuring performance and scalability.
• Create and manage ETL/ELT pipelines for large-scale data ingestion and transformation.

Streaming & Event Processing

• Build and manage real-time streaming systems using Spark Streaming or Storm.
• Work with Kafka / RabbitMQ for event-driven ingestion and messaging patterns.

Cloud & Databricks Engineering

• Develop & optimize workloads on AWS Databricks or Azure Databricks.
• Perform cluster management, job scheduling, performance tuning, and automation.

Data Integration & Storage

• Integrate data from diverse sources: RDBMS (Oracle, SQL Server), ERP, file systems.
• Work with query engines like Hive and Impala.
• Experience with NoSQL stores: HBase, Cassandra, MongoDB.

Programming & Scripting

• Strong hands-on coding in Python for data transformations and automations.
• Strong SQL skills for data validation, tuning, and complex queries

Team Leadership (L4)

• Provide technical leadership and mentoring to junior engineers.
• Drive solution design for Big Data platforms end-to-end

Ways of Working

• Work in Agile teams, participate in sprint ceremonies and planning.
• Collaborate with engineering, data science, and product teams.

Required Skills & Expertise (Both L2 & L4)

• Apache Spark – Expert level (core, SQL, streaming)
• Python – Strong hands-on
• Distributed computing fundamentals
• Hadoop ecosystem: Hadoop v2, MapReduce, HDFS, Sqoop
• Streaming systems: Spark Streaming / Storm
• Messaging: Kafka or RabbitMQ
• SQL – Advanced (joins, stored procedures, query optimization)
• NoSQL: HBase, Cassandra, MongoDB
• ETL frameworks & data pipeline design
• Hive / Impala querying
• Performance tuning of Spark jobs
• AWS or Azure Databricks
• Experience working in Agile

Experience & Level Mapping

L2 – Mid-Level (4–5 Yrs)

• Skills: Spark, Python, AWS
• Notice Period: Immediate – 20 Days
• CTC Band: Up to 21 LPA

L4 – Senior-Level (8–12 Yrs)

• Skills: Spark, Python, Azure Databricks
• Notice Period: 15 Days (Nov joiners) OR Jan joiners
• CTC Band: Up to 42 LPA