Azure & Databricks - Senior Data Engineer

Why Valtech? We’re advisors, visionaries, creative and techies. We embrace all things digital. We talk to each other. We have fun. We love our clients. We’re looking ahead • We are global

Join our innovative, fast-growing data team at the forefront of cloud data architecture on Microsoft Azure. We're building scalable, secure, and modern data platforms using cutting-edge Azure services and Databricks unified analytics platform. If you're passionate about creating high-performance data infrastructure and solving complex big data challenges in a cloud-native environment, this is the perfect opportunity for you.

As a Senior Data Engineer specializing in Azure and Databricks, you will architect and implement enterprise-grade cloud-native data solutions on the Microsoft Azure ecosystem. This is a hands-on engineering role with significant architectural influence, where you'll work extensively with Azure Data Factory, Databricks, Delta Lake, and other modern data tools to create efficient, maintainable, and scalable data pipelines using medallion architecture and lakehouse patterns.

What you do

4–7 years of hands-on experience in data engineering with strong focus on cloud data platforms. Proven experience with big data technologies and distributed computing

Technical Expertise

Azure Mastery: Deep expertise in Azure data services including Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure SQL Database

Databricks Proficiency: Advanced proficiency in Databricks including cluster management, notebook development, and Delta Lake

Big Data Processing: Strong experience with Apache Spark (PySpark, Scala), distributed computing concepts, and performance optimization

Programming: Proficiency in Python, Scala, and SQL for data processing and transformation

Data Architecture

Experience with lakehouse architecture, medallion architecture, and modern data warehouse design patterns

Streaming Technologies: Knowledge of real-time data processing using Azure Event Hubs, Kafka, and Spark Streaming

Cloud Platforms: Deep understanding of Microsoft Azure ecosystem and native data services

Data Management & DevOps

Data Governance: Understanding of data governance frameworks, Unity Catalog, and data quality practices

DevOps: Experience with CI/CD practices, Git-based workflows, and Infrastructure as Code

Orchestration: Knowledge of data orchestration tools like Azure Data Factory, Airflow, or similar platforms

Security: Understanding of Azure security best practices, RBAC, and data encryption

Advanced Technologies

Experience with Azure Machine Learning and MLOps practices

Knowledge of containerization (Docker, Kubernetes) and Azure Container Instances

Experience with Power BI for data visualization and reporting

Understanding of data mesh architecture and domain-driven design principles

Experience with Azure DevOps for CI/CD pipeline management

Familiarity with monitoring tools like Azure Monitor, Application Insights, and Databricks monitoring

Specialized Skills

Experience with real-time analytics and complex event processing

Knowledge of graph databases and Azure Cosmos DB

Understanding of data lake optimization techniques and performance tuning

Experience with multi-cloud or hybrid cloud architectures

Technology Stack

Core Azure Services

Data Storage: Azure Data Lake Storage Gen2, Azure Blob Storage, Azure SQL Database

Data Processing: Databricks, Azure Synapse Analytics, Azure Data Factory

Streaming: Azure Event Hubs, Azure Stream Analytics, Kafka

Analytics: Databricks SQL, Azure Analysis Services, Power BI

Machine Learning: Azure Machine Learning, MLflow, Databricks ML

Big Data & Processing

Compute: Apache Spark (PySpark, Scala), Databricks Runtime

Data Formats: Delta Lake, Parquet, JSON, Avro

Architecture: Medallion Architecture, Lakehouse, Data Mesh

Orchestration: Azure Data Factory, Airflow, Azure Logic Apps

Development & DevOps

Programming: Python, Scala, SQL, PowerShell

Version Control: Git, Azure DevOps, GitHub

Infrastructure: ARM Templates, Terraform, Azure Resource Manager

Monitoring: Azure Monitor, Application Insights, Databricks monitoring

Security & Governance

Security: Azure Active Directory, Key Vault, RBAC

Governance: Unity Catalog, Azure Purview, Data Lineage

Compliance: GDPR, data encryption, access controls

Certifications

Microsoft Azure Data Engineer Associate (DP-203)

Databricks Certified Associate Developer for Apache Spark

Databricks Certified Professional Data Engineer

Azure Solutions Architect or Azure Data Fundamentals certifications

What we ask

Data Architecture & Engineering

Design and implement enterprise-scale, robust data solutions using Azure and Databricks

Architect scalable ELT/ETL pipelines using Azure Data Factory, Azure Synapse Analytics, and Databricks

Build automated data ingestion processes from various sources including streaming and batch data

Develop and maintain data transformation workflows using PySpark, Scala, and SQL on Databricks

Data Platform & Optimization

Implement lakehouse architecture using Delta Lake and Databricks Delta tables

Design and optimize Databricks clusters for performance, cost management, and resource utilization

Apply medallion architecture (Bronze, Silver, Gold) for data processing and transformation

Optimize Spark jobs and queries for maximum performance and cost efficiency

Big Data & Analytics

Build streaming data pipelines using Azure Event Hubs, Kafka, and Databricks Structured Streaming

Implement machine learning pipelines using MLflow and Databricks ML capabilities

Design and maintain data models for analytical workloads and real-time processing

Work with large-scale datasets and implement partitioning strategies for optimal performance

Quality & Governance

Develop comprehensive data quality tests and monitoring using Great Expectations and custom frameworks

Implement data governance, lineage, and security policies within Azure and Databricks

Create and maintain comprehensive data documentation and lineage tracking using Unity Catalog

Build data validation and testing frameworks for proactive data monitoring

DevOps & Automation

Build and maintain CI/CD pipelines for Databricks notebooks and data workflows

Develop custom Python/Scala scripts for data processing, manipulation, and automation tasks

Implement Infrastructure as Code (IaC) using ARM templates or Terraform for Azure resources

Work with orchestration tools for pipeline scheduling and dependency management

Collaboration & Analytics

Collaborate with data scientists, analysts, and ML engineers to support advanced analytics use cases

Enable self-service analytics capabilities using Databricks SQL and Azure Analytics services

Work closely with stakeholders to understand data requirements and deliver scalable solutions

What we offer

You’ll join an international network of data professionals within our organisation. We support continuous development through our dedicated Academy. If you're looking to push the boundaries of innovation and creativity in a culture that values freedom and responsibility, we encourage you to apply.

At Valtech, we’re here to engineer experiences that work and reach every single person. To do this, we are proactive about creating workplaces that work for every person at Valtech. Our goal is to create an equitable workplace which gives people from all backgrounds the support they need to thrive, grow and meet their goals (whatever they may be). You can find out more about what we’re doing to create a Valtech for everyone here.

Please do not worry if you do not meet all of the criteria or if you have some gaps in your CV. We’d love to hear from you and see if you’re our next member of the Valtech team!