Senior Data Engineer - Databricks:
On behalf of our Financial Services client, Procom is searching for a Senior Data Engineer - Databricks for a 12 month contract role. This position is a hybrid position with 2 days onsite at our client’s Toronto, ON office.
Senior Data Engineer - Databricks - Job Description:
This role involves integrating data from diverse sources into the Enterprise Data Platform using Databricks, Snowflake, and AWS. The project focuses on developing and optimizing data ingestion and streaming pipelines, as well as migrating to Snowflake. The successful candidate will work collaboratively to enhance data value and contribute to software engineering best practices.
Senior Data Engineer - Databricks - Responsibilities:
Design and implement batch and streaming ingestion pipelines on Databricks.
Model and manage datasets using Apache Iceberg tables.
Implement end-to-end data governance with Unity Catalog.
Build robust, observable pipelines and ensure secure, compliant data movement.
Optimize cost and performance of data processing.
Contribute to best practices, standards, and reusable components.
Senior Data Engineer - Databricks - Mandatory Skills:
Bachelor’s degree in IT or equivalent experience.
At least 10 years of experience with data integration and ETL pipeline development.
Minimum 5 years of experience with Databricks and Apache Spark in cloud environments.
Strong understanding of Apache Iceberg and data governance with Unity Catalog.
Proficiency in orchestrating and scheduling pipelines.
Solid SQL and data modeling skills.
Senior Data Engineer - Databricks – Nice-to-Have Skills:
Experience with Snowpipe and IICS API for real-time data ingestion.
Knowledge of ETL techniques and data ingestion patterns.
Familiarity with AWS services like S3, EC2, and CloudWatch.
Experience with DevOps and CI/CD tools like GitHub Actions.
Strong software engineering practices.
Senior Data Engineer - Databricks – Assignment Length:
This is a 12 month contract position.
Senior Data Engineer - Databricks - Start Date:
ASAP.
Senior Data Engineer - Databricks - Assignment Location:
Toronto, ON, Canada. This position is hybrid with 2 days onsite.