The Senior Data Engineer plays a criticalrole in designing, developing, and maintaining scalable data infrastructure and pipelines. This position supports data-driven decision-making across the organization by ensuring data quality, accessibility, and performance.
Roles and Responsibilities
Data Pipeline Development:
Design, build, and maintain robust, scalable data pipelines using tools like Apache Airflow, Kafka, and Spark.
Data Architecture:
Develop and optimizedata models, data warehouses, and ETL processes to support analytics and machine learning initiatives.
Collaboration:
Work closelywith data scientists, analysts, and software engineers to understand data requirements and deliver tailored solutions.
Data ǪualityG Governance:
Ensure data integrity, implement validation checks, and enforce data governance policies.
Performance Optimization:
Monitor and improvedata processing speed,system uptime, and query performance.
Leadership G Mentorship:
Guide junior engineers, promote best practices, and foster a culture of continuous learning.
Innovation G Research:
Stay updated with emerging technologies in AI, automation, and data streaming; lead initiatives to integrate them into existing systems.
Profile
Bachelor's or Master's in computer science, data science, information technology, or a related field, with
6+ years of experience
.
Proficiency in
SǪL
,
Python
, and
ETL/ELT tools
Experience in Microsoft solutions are as follows:
Azure Data Factory
Microsoft Fabric
Azure Synapse Analytics
Azure Databricks
Power BI
Azure SǪL Database/ SǪL Server
Azure Purview
Experience with
big data technologies
(Hadoop, Spark)
Familiarity with
cloud platforms
(Azure, AWS, and GCP)
Strong understanding of
data modelling
,
data warehousing
, and
stream processing
Knowledge of
data security
,
compliance
, and
governance
Knowledge of
CI/CD pipelines
using Azure DevOps
Understanding of
data security and compliance
(e.g., GDPR, HIPAA)