Job Profile: Sr Data Engineer
Job Type: Long-time Contract Job Opportunity
Location: 100% Remote in Mexico
Job Description:
We are searching for a skilled Data Engineer to join our team. The ideal candidate will have hands-on experience in development and production implementation with Azure Databricks and Spark, proficiency in Python (PySpark) and SQL coding, and a strong understanding of data lake architecture, ETL processes, and data quality management. Experience with Azure data analytics services and excellent technical writing skills are also required.
**Responsibilities:**
- Develop and implement data processing pipelines using Databricks and Spark.
- Write efficient and optimized code in Python (PySpark) and SQL for TB-scale data processing.
- Design and maintain data lake architecture and ETL processes.
- Ensure data quality and integrity through robust data quality management practices.
- Utilize Azure data analytics services for data storage, processing, and analysis.
- Create comprehensive technical documentation, including requirements and testing documents.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Maintain version control using Git.
**Requirements:**
- 5+ years of hands-on experience in development and production implementation with Azure Databricks and Spark.
- Proficiency in Python (PySpark) and SQL coding.
- Proven ability to optimize performance for TB-scale data processing.
- Strong understanding of data lake architecture, ETL processes, and data quality management.
- Experience with Azure data analytics services.
- Excellent technical writing skills for creating requirements and testing documents.
- Ability to communicate technical concepts clearly and concisely.
- Knowledge of Git.