We are looking for a Lead Data DevOps Engineer to drive the migration of Spark, Flink, and additional services to Databricks.
You will be responsible for creating and managing robust cloud infrastructure to enhance data processing pipelines. Join us to leverage your AWS, Terraform, and Databricks expertise and help deliver cutting-edge data solutions.
Responsibilities
Develop, launch, and oversee Databricks infrastructure with Terraform to support scalable data operations
Enhance Spark and Flink workflows and coordinate job orchestration for optimal pipeline performance within Databricks
Architect, implement, and sustain AWS cloud infrastructure ensuring high availability and top performance
Work closely with stakeholders to collect requirements and execute infrastructure upgrades aligned with business goals
Track and resolve issues in data infrastructure using monitoring tools like Datadog to guarantee uptime
Refine and stabilize CI/CD pipelines leveraging Spinnaker and Jenkins for seamless continuous delivery
Maintain adherence to security policies and best practices across the data infrastructure
Requirements
Demonstrated track record in Data DevOps engineering with over 5 years managing cloud infrastructure
Advanced skills in Amazon Web Services (AWS) cloud environment
Proficient in managing Databricks platforms and associated data workflows
Experienced in using Terraform for infrastructure as code deployment and management
Comprehensive understanding of Spark and Flink data processing systems
Capability to enhance CI/CD pipelines and automate deployment workflows
Effective communication skills to collaborate within cross-functional teams
Upper-Intermediate English proficiency (B2 level) for effective interaction with international teams
We offer
Career plan and real growth opportunities
Unlimited access to LinkedIn learning solutions
Constant training, mentoring, online corporate courses, eLearning and more
English classes with a certified teacher
Support for employee’s initiatives (Algorithms club, toastmasters, agile club and more)
Enjoyable working environment (Gaming room, napping area, amenities, events, sport teams and more)
Flexible work schedule and dress code
Collaborate in a multicultural environment and share best practices from around the globe
Hired directly by EPAM \& 100% under payroll
Law benefits (IMSS, INFONAVIT, 25% vacation bonus)
Major medical expenses insurance: Life, Major medical expenses with dental \& visual coverage (for the employee and direct family members)
13 % employee savings fund, capped to the law limit
Grocery coupons
30 days December bonus
Employee Stock Purchase Plan
12 vacations days
Official Mexican holidays, plus 5 extra holidays (Maundry Thursday and Friday, November 2nd, December 24th \& 31st)
Monthly non-taxable amount for the electricity and internet bills
EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.
By applying to our role, you are agreeing that your personal data may be used as in set out in EPAM´s Privacy Notice and Policy.