Data Engineer - Data Bricks

Role & responsibilities

• Design and develop end-to-end data pipelines using Azure Data Factory (ADF)
• Build and maintain data processing workflows using Azure Databricks (ADB) notebooks
• Develop and manage batch processing jobs using Spring Batch framework
• Work extensively with PostgreSQL databases for querying, data manipulation, and performance optimization
• Perform data transformations, validations, and aggregations within pipelines and notebooks
• Monitor, troubleshoot, and optimize data pipelines, batch jobs, and workflows
• Ensure data quality, integrity, and consistency across multiple systems
• Collaborate with stakeholders to gather requirements and deliver data solutions
• Handle production issues and ensure minimal downtime of data systems

Required Skills & Qualifications

Technical Skills

• Hands-on experience with Spring Batch
• Proficiency in PostgreSQL (complex queries, joins, indexing, optimization)
• Experience with Azure Data Factory (pipeline creation, triggers, integrations)
• Experience with Azure Databricks (notebooks, data processing)
• Solid understanding of data pipeline architecture and ETL/ELT processes

Good to Have

•

• Experience with PySpark / Spark

•

• Familiarity with cloud concepts and distributed systems

•

• Exposure to CI/CD pipelines and version control (Git)

Key Competencies

• Strong problem-solving and analytical skills
• Ability to independently own and deliver end-to-end solutions
• Debugging and troubleshooting mindset