MAIN OBJECTIVE OF ROLE
To design, build, and maintain scalable data architectures and pipelines using AWS services, focusing on efficiently ingesting, processing, and transforming data to ensure high quality and accessibility for analytics and reporting.
Key Responsibilities
Designs, builds and maintains scalable data pipelines to ingest and process data from various sources.
Leverages Amazon Web Services (AWS) tools like AWS Glue, Amazon S3, Amazon Redshift, and Amazon RDS to manage data pipeline, storage, transformation, and analytics.
Builds and maintains Cloud framework for orchestrating data-pipelines, continuously optimizing and enhancing it to integrate new data sources effectively.
Develops Extract, Transform, Load (ETL) processes to ensure data is cleaned, transformed, and loaded into data warehouses. Use appropriate scripting such as PySpark, Python, Scala to perform data transformations.
Automates data integration and processing tasks to enhance efficiency and reduce manual intervention.
Sets up monitoring tools to track data pipeline performance and troubleshoot issues as they arise.
Maintains comprehensive documentation of data architectures, processes, and workflows, ensuring updates are made consistently with every release.
Follows defined Data Governance policies and guidelines and enforces team to align with standard procedures.
Ensures data security and compliance with relevant regulations, including GDPR or industry-specific guidelines.
Leads the junior data engineers, provide technical guidance, manage work allocation and delivery.
Leads and mentors junior data analysts, collaborate with cross-functional teams to meet data needs, implements data governance practices, and optimizes performance to support informed decision-making within the organization.
Qualifications
Bachelor's Degree (3+ years)
Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field
Fluent in English
Must have 8+ years of industry experience, with 5 years working with design and development of data pipelines at enterprise scale. Hands on experience in building streaming, batch pipelines for structured, semi-structured data Must have, solid experience in handling large volume of data and performing transformation using scripts such as PySpark, Scala, Python. Must have experience working with AWS cloud services including EMR, Lambda, MKS, Redshift, GLue, Preference for working experience in airline industry
AWS EMR, MKS, Glue, PySpark, Python, Scala, OpenSearch, Redshift, Dynamo DB, RDS
Years with qualifications: 8 - 10 years
COMPETENCIES
Customer Focus
Teamwork
Effective Communication
Personal Accountability \& Commitment to achieve
Resilience \& Flexibility (Can do attitude)
Isr Requirements
Reads and complies with the ISR policies of the Company and diligently reports any weakness or incidents to the respective Line Manager or the Information Security team. Completes all required ISR awareness sessions and follows associated guidelines in the day-to-day business operations.