The DevOps Engineer plays a critical role in enabling scalable, reliable, and automated data solutions across the Azure ecosystem. This position focuses on supporting end-to-end data and ML pipelines, primarily built on Azure
DevOps, Azure Databricks, and modern Infrastructure-as-Code (IaC) practices using Bicep or Terraform.
The candidate collaborates closely with our client's Data Engineers, Security Team and Platform Teams to ensure smooth development workflows, automated deployments, and securely governed cloud environments all according to our client's Quality standards.
Key Responsibilities
CI/CD Pipeline Engineering (Azure DevOps)
Design, develop, and maintain Azure DevOps pipelines for data
processing workflows, ML model training, and Databricks deployments.
Implement pipeline quality gates, automated testing, environment
promotion strategies, and artifact management.
Ensure pipelines are resilient, observable, and aligned with
organizational standards.
Ensure the Software Development Lifecycle is built up according to the
client's Policies and meet their requirements.
Infrastructure as Code (IaC)
Build, maintain, and standardize cloud infrastructure using Bicep or Terraform for Azure resources such as Databricks Workspaces, Storage Accounts, Key Vaults, Networks and containerized where applicable
Ensure infrastructure is modular, reusable, and compliant with enterprise security and governance requirements.
Automation \& Scripting (PowerShell, Python)
Develop automation for recurring operational tasks (orchestration, monitoring, environment provisioning) using PowerShell and Python.
Create scripts supporting Databricks job deployments, cluster lifecycle management, ML model registration, and data workflow automation.
Databricks \& Machine Learning Operations (MLOps)
Build and maintain deployment mechanisms for Databricks notebooks/Jobs, workflows, ML models, and Delta pipelines.Support ML lifecycle automation, including data validation, model packaging, model registry updates, and automated retraining pipelines.- Collaborate with Data Science teams to operationalize machine learning workflows in production environments.
Cloud Environment Management \& Reliability
Ensure high availability, scalability, and reliability of data and ML workloads in Azure.
Implement monitoring \& alerting for pipelines, clusters, and data workflows.
Contribute to operational improvements and proactive issue prevention.
Collaboration \& Governance
Work closely with cross-functional teams and participate in review processes for IaC, pipeline changes, and data platform enhancements.
Ensure changes follow standardized processes as outlined in the client's Quality Handbook
Document designs, processes, and architecture diagrams to support transparency and long-term maintainability.
Required Skills \& Experience
Technical Skills
Azure DevOps pipelines (YAML), environments, approvals, artifacts
Infrastructure as Code: Bicep and/or Terraform and/or Pulumi Scripting Languages: PowerShell and Python
Strong understanding of Azure Databricks, Spark fundamentals, and Databricks deployment patterns
Experience with Azure Core Services: Key Vault, Storage, VNet, AAD, Monitor, AKS (optional)
Familiarity with containerization, Git branching strategies, and DevOps best practices
Experience with MLOps frameworks (MLflow, Databricks Model Registry)
Experience deploying large-scale data or ML workloads
Knowledge of cloud security best practices and networking in Azure
Soft Skills
Strong communication and ability to collaborate across data, engineering, and infrastructure teams
Analytical mindset with a passion for automation and continuous improvement
Ability to troubleshoot complex distributed systems and data pipelines
Additional Information
Rate offered: £450-475 per day
IR35 Status: Outside
Location: Remote
Start date: March '26
Duration: 3 months initial sign up with significant opportunity for extension.