Data Engineer (Hybrid – London, UK)
Overview
An opportunity for a skilled and motivated
Data Engineer
to join a growing data engineering team within a fast-paced and innovative environment. This position is ideal for someone who enjoys designing robust, scalable data infrastructure using modern open-source technologies. The role offers exposure to diverse datasets, complex data challenges, and opportunities to influence technical direction and best practices.
Key Responsibilities
Design, develop, and maintain
data ingestion pipelines
using open-source frameworks and tools.
Build and optimise
ETL/ELT processes
for small to large-scale data processing requirements.
Develop
data models
and
schemas
to support analytics, business intelligence, and product needs.
Monitor, troubleshoot, and optimise
data pipeline performance and reliability
.
Collaborate with analysts, engineers, and product teams to gather and translate data requirements.
Implement
data validation
and
quality checks
to ensure data integrity.
Participate in
architecture discussions
and contribute to
technical roadmap planning
.
Skills and Attributes
Strong analytical mindset with exceptional attention to detail.
Excellent
problem-solving
and
debugging
skills.
Ability to
work independently
and manage multiple priorities effectively.
Clear and confident
communication skills
, with the ability to explain technical concepts to non-technical stakeholders.
Experience working in
agile development environments
.
Passion for
continuous learning
and staying current with emerging data technologies.
Technical Requirements
Proficiency in
SQL
, including complex query design and optimisation.
Strong
Python
programming skills, particularly with libraries such as
pandas
,
NumPy
, and
Apache Spark
.
Experience building and maintaining
data ingestion pipelines
and optimising performance.
Hands-on experience with open-source data frameworks such as
Apache Spark
,
Apache Kafka
, or
Apache Airflow
.
Knowledge of
distributed computing
and
big data
concepts.
Experience using
version control systems
(Git) and
CI/CD
practices.
Familiarity with
relational databases
(PostgreSQL, MySQL, or similar).
Experience with
containerisation
technologies (
Docker
,
Kubernetes
).
Understanding of
data orchestration tools
(e.g., Airflow or Dagster).
Knowledge of
data warehousing
principles and
dimensional modelling
.
Understanding of
cloud platforms
and
infrastructure-as-code (IaC)
practices.
Awareness of
real-time streaming
and
data quality monitoring
solutions.
Preferred Qualifications
Bachelor’s degree in
Computer Science
,
Engineering
,
Mathematics
, or a related field.
2–5 years
of experience in
data engineering
or similar technical roles.
Familiarity with data processes and challenges in the
energy
or
commodities
sectors (beneficial but not required).