About the Role
We are seeking a highly skilled
Big Data \& DevOps Engineer
to design, build, and maintain scalable, reliable big data platforms while driving automation and standardization across development and operations. As a key member of our team, you will bridge the gap between data engineering and DevOps, ensuring seamless data pipelines, efficient CI/CD workflows, and robust infrastructure management to support data-driven business growth.
Responsibilities
Big Data Platform Development \& Operations
Architect and deploy distributed big data ecosystems (e.g., Hadoop, Spark, Flink, Kafka) for real-time and batch data processing, ensuring high availability and performance at scale.
Optimize data pipelines for ingestion, transformation, and storage, leveraging tools like NiFi, Airflow, or Luigi to automate workflows.
Manage cluster resources and monitor platform health using tools like YARN, Prometheus, Grafana, or Datadog, resolving issues to maintain 99.9% uptime.
DevOps Engineering \& Automation
Implement end-to-end CI/CD pipelines using Jenkins, GitLab CI/CD, or GitHub Actions, ensuring automated testing, deployment, and rollback capabilities.
Design and manage infrastructure as code (IaC) with Terraform, Ansible, or CloudFormation across public clouds (AWS/Azure/GCP) and on-premises environments.
Develop and enforce DevOps best practices, including logging, monitoring, backup strategies, and disaster recovery plans to enhance system observability and resilience.
Cross-Functional Collaboration
Partner with data engineers, data scientists, and software developers to understand requirements and deliver scalable solutions for data processing and analytics.
Participate in technical design reviews, advocating for cloud-native, containerized architectures (e.g., Docker, Kubernetes) and microservices.
Train and mentor teams on DevOps methodologies, fostering a culture of collaboration and continuous improvement.
Security \& Compliance
Ensure data platforms adhere to security standards (e.g., GDPR, HIPAA) through encryption, IAM policies, and vulnerability scanning.
Monitor and mitigate security risks, such as data breaches or unauthorized access, implementing proactive threat detection and incident response protocols.
Required Skills
At least 2 years experiences, experienced with Hive, Hadoop, Spark, Python;
Good knowledge and experience with Linux;
Experience with containerization technologies such as Docker and Kubernetes.
Familiarity with DevOps best practices and tools such as Git, Jenkins
Good learning attitude;
Good English speaking and writing.