About The Role
About the Role
We have an exciting opportunity for a skilled Databricks Data Engineer with deep PySpark expertise and hands-on experience in building pipelines across Databricks and Azure Data Factory.
The role will be based out of our Bournemouth office and is a fixed-term contract for 12 months.
You will design and deliver robust, scalable, and governed data flows that form the backbone of our Lakehouse platform, ensuring high-quality, trusted data for analytics, reporting, and regulatory requirements.
Key Responsibilities
Build and maintain Databricks pipelines (batch and incremental) using PySpark and SQL.
Orchestrate end-to-end data workflows with Azure Data Factory.
Develop and optimise Delta Lake tables (partitioning, schema evolution, vacuuming).
Implement the Medallion Architecture (Bronze, Silver, Gold) to transform raw data into business-ready datasets.
Apply row-count checks, logging, and error-handling frameworks for pipeline monitoring and governance.
Integrate Databricks pipelines with downstream systems including Power BI and reporting platforms.
Collaborate with analysts, business teams, and other engineers to deliver consistent, well-documented datasets.
Support deployments and automation via Azure DevOps CI/CD pipelines.
Gather and refine requirements from business stakeholders.
About You
Strong PySpark development skills for large-scale data engineering.
Proven experience with Databricks pipelines and workflow management.
Expertise in Azure Data Factory for orchestration and scheduling.
Solid knowledge of Delta Lake and Lakehouse data modelling principles.
Hands-on experience with SQL for data transformation and validation.
Familiarity with Azure services (ADLS/Blob, Key Vault, SQL).
Knowledge of ETL/ELT frameworks, error logging, and monitoring.
Desirable skills
Understanding of Microsoft Fabric.
Experience with DevOps practices (CI/CD, release pipelines).
Awareness of data governance and lineage tools.
Experience working in the Finance sector.
Rewards \& Benefits
This role is a Band C in the LV\= Structure.
At LV\= Life and Pensions, you’ll go above and beyond to do the right thing for our customers. We’ll reward your hard work with an attractive, competitive salary and benefits package, which includes:
30 days' holiday, with the option to buy up to 2 additional days
Competitive pension scheme - LV\= Life and Pensions will double-match the amount you pay, up to 14% (subject to National Minimum Wage requirements)
An annual bonus scheme based on company and personal performance
Single-cover private medical insurance (with the option for you to upgrade to family cover)
Flexible benefits, including a cycle to work scheme, personal accident insurance, critical illness cover and dental insurance
Up to 20% discount on our life products for you and your immediate family
A group life assurance policy with 4 x your basic pay to go to your dependents (you’ll have the option to increase to 8 x cover)
Group Income Protection (if you become a member of the Pension scheme and reach 5 years of service)
Access to our Employee Assistance Programme (EAP) for support when you need it
A virtual GP service
Shared parental leave
About Us
We’re proud of our inclusive culture at LV\= and, as an equal-opportunity employer, we continually work to remove unconscious bias from our recruitment process. We value our colleagues for what they bring to our team regardless of any protected status or characteristics they may have. Talk to us about flexible working as part of your application; if it’s right for you, our members and customers, and our business, then we’ll do everything we can to make it happen.
Please note that we are unable to offer Skilled Worker Visa Sponsorship for this role. Therefore, you must ensure that you are eligible to work in the UK without our sponsorship for your application to be considered.
Any successful internal colleagues will be offered a secondment opportunity. The secondment policy is available on the intranet for internal colleagues.