We are looking for a Lead Data DevOps Engineer to drive the migration of Spark, Flink, and additional services to Databricks.
You will be responsible for creating and managing robust cloud infrastructure to enhance data processing pipelines. Join us to leverage your AWS, Terraform, and Databricks expertise and help deliver cutting-edge data solutions.
Responsibilities
Develop, launch, and oversee Databricks infrastructure with Terraform to support scalable data operations
Enhance Spark and Flink workflows and coordinate job orchestration for optimal pipeline performance within Databricks
Architect, implement, and sustain AWS cloud infrastructure ensuring high availability and top performance
Work closely with stakeholders to collect requirements and execute infrastructure upgrades aligned with business goals
Track and resolve issues in data infrastructure using monitoring tools like Datadog to guarantee uptime
Refine and stabilize CI/CD pipelines leveraging Spinnaker and Jenkins for seamless continuous delivery
Maintain adherence to security policies and best practices across the data infrastructure
Requirements
Demonstrated track record in Data DevOps engineering with over 5 years managing cloud infrastructure
Advanced skills in Amazon Web Services (AWS) cloud environment
Proficient in managing Databricks platforms and associated data workflows
Experienced in using Terraform for infrastructure as code deployment and management
Comprehensive understanding of Spark and Flink data processing systems
Capability to enhance CI/CD pipelines and automate deployment workflows
Effective communication skills to collaborate within cross-functional teams
Upper-Intermediate English proficiency (B2 level) for effective interaction with international teams