Lead Data Engineer (Contract)
Dublin - Hybrid
12 Month Contract (Potential Extension)
I’m working with a leading organisation seeking a Lead Data Engineer (Contract) to help design and evolve a cutting-edge Databricks + AWS Lakehouse. This role is pivotal in enabling teams to derive insight into complex behavioural patterns and act decisively.
Key Responsibilities:
Lead the end-to-end design, build, and optimisation of scalable Spark / PySpark pipelines (batch + streaming) on Databricks
Define and enforce lakehouse / Medallion architecture standards (bronze, silver, gold), schema governance, and quality SLAs
Architect secure, compliant AWS data infrastructure (S3, Glue, IAM, KMS, Lake Formation)
Implement orchestration with Airflow, Databricks Workflows, and Step Functions
Drive CI/CD, infrastructure-as-code, and data lineage / governance (Unity Catalog, OpenLineage)
Mentor engineers, lead reviews, and ensure high performance and cost efficiency
About You:
Expert in Databricks, Snowflake, Python, and AWS
Proven experience building and optimising large-scale data solutions
Strong leadership, stakeholder collaboration, and delivery focus
Experience in secure, governed, cloud-first environments
Contract: 12 months \| Hybrid \| Start: ASAP