Position Summary:
Data Engineer for data migration projects (Primarily utilizing AWS, IDMC, Databricks and Tableau)
Responsibilities:
Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and enrich data, making it suitable for analytical purposes using Databricks' Spark capabilities and Informatica Power Center/IDMC for data transformation and quality.
Monitor and optimize data processing and query performance in both AWS and Databricks environments, making necessary adjustments to meet performance and scalability requirements. Utilize Informatica Power Center/IDMC for optimizing data workflows.
Implement security best practices and data encryption methods to protect sensitive data in both AWS and Databricks, while ensuring compliance with data privacy regulations. Employ Informatica IDMC for data governance and compliance.
Implement automation for routine tasks, such as data ingestion, transformation, and monitoring, using AWS services like AWS Step Functions, AWS Lambda, Databricks Jobs, and Informatica IDMC for workflow automation.
Maintain clear and comprehensive documentation of data infrastructure, pipelines, and configurations in both AWS and Databricks environments, with metadata management facilitated by Informatica IDMC.
Identify and resolve data-related issues and provide support to ensure data availability and integrity in both AWS, Databricks, and Informatica Power Center/IDMC environments.
Create, manage, and optimize data visualization process using as Tableau, OAS or Power BI
Stay up-to-date with AWS, Databricks, Informatica Power Center/IDMC services, and data engineering best practices to recommend and implement new technologies and techniques.
Requirements:
Bachelor’s or master’s degree in computer science, data engineering, or a related field.
Minimum 4 years of experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica Power Center/IDMC.
Proficiency in programming languages such as Python for building data pipelines.
Evaluate potential technical solutions and make recommendations to resolve data issues especially on performance assessment for complex data transformations and long running data processes.
Strong knowledge of SQL databases.
Familiarity with data modeling and schema design.
AWS certifications: (e.g., AWS Certified Data Analytics - Specialty, AWS Certified Data Analytics - Specialty), Databricks certifications, and Informatica certifications are a plus.