Role Overview
We are seeking an experienced
Data Engineer
to design, develop, and implement end-to-end data pipelines and analytics solutions that enable data-driven decision-making. The ideal candidate will work across multiple workstreams, supporting data ingestion, transformation, and visualization requirements while ensuring adherence to data quality and governance standards.
Key Responsibilities
Collaborate across workstreams to understand and support data requirements for reports, dashboards, and analytical initiatives.
Analyze and perform data profiling to identify patterns, discrepancies, and quality issues, ensuring compliance with
Data Quality
and
Data Management
processes.
Design and develop
end-to-end (E2E) data pipelines
, including data ingestion, transformation, processing, and surfacing for large-scale applications.
Develop and automate data pipelines using
Azure
or
AWS
data platforms and technology stacks such as
Databricks
,
Data Factory
, and
Spark
.
Translate business requirements into clear technical specifications to guide system analysts and technical teams.
Perform
data ingestion
in both
batch and real-time
modes using methods such as file transfer,
APIs
,
Kafka
, and
Spark Streaming
.
Build and optimize
ETL processes
using
Spark
and related frameworks for data transformation and standardization.
Design and develop
data visualization
solutions using
Power BI
,
Tableau
, or similar tools, including
data export
and
API
integration where required.
Support data analysis and model development to improve accessibility, scalability, and quality of data for business users.
Technical Skills:
Proficiency in
Python
,
SQL
, and
Apache Spark
for data processing.
Hands-on experience with
Azure
and/or
AWS
data platforms.
Strong understanding of
Databricks
,
Data Factory
,
ETL
, and
API
development.
Exposure to
data architecture
and
solution architecture
principles.
Knowledge of
data lakes
,
data warehousing
,
big data tools
,
RDBMS
,
NoSQL
, and
Knowledge Graphs
.
Data Visualization Skills:
Experience with
Power BI
or equivalent visualization tools.
Knowledge of
DAX programming
,
data modeling
, and
storytelling
through visual design.
Business Analysis Skills:
Strong analytical mindset with the ability to interpret business requirements.
Experience in
data profiling
,
basic data model design
, and
requirement analysis
.
Proficient in
SQL
for business data analysis.