Database Engineer

Key Responsibilities

• Develop and maintain scalable data pipelines for batch and real-time data processing
• Build robust data ingestion frameworks using APIs, file-based ingestion, and streaming sources
• Design and implement Python and PySpark-based data transformation logic for large-scale datasets
• Develop Python-based microservices and REST APIs for secure and efficient data access and sharing
• Integrate data pipelines with platforms such as Databricks, Delta Lake, and Snowflake
• Experience with serverless data processing (e.g., AWS Lambda, Azure Functions with Python
• Implement and manage streaming pipelines using Kafka / Kinesis
• Work with structured, semi-structured, and unstructured data (Parquet, JSON, CSV, sensor/IoT data
• Optimize data processing jobs for performance, scalability, and cost efficiency
• Apply best practices for data partitioning, storage optimization, and query performance tuning
• Ensure data quality, validation, and monitoring across pipelines
• Collaborate with data architects, analysts, and downstream consumers to deliver reliable datasets
• Support data modeling and transformation aligned with Lakehouse principles (e.g., medallion architecture)
• Contribute to automation and deployment of pipelines using orchestration and CI/CD practices

Required Skills & Experience

• 2–4 years of hands-on experience in Data Engineering
• Strong programming expertise in Python and PySpark
• Solid experience in SQL for data transformation and analysis
• Hands-on experience building data pipelines (ETL/ELT) in production environments
• Experience working with APIs and microservices for data ingestion and exposure
• Strong understanding of data processing frameworks such as Spark, AWS Glue, or Dbt
• Experience handling large-scale datasets across multiple formats (Parquet, JSON, CSV, etc)
• Good understanding of Data Lake / Lakehouse concepts and implementation patterns