Job Summary
We are seeking a highly skilled Sr. Developer with 7 to 11 years of experience to join our dynamic team. The ideal candidate will have expertise in Spark in Scala Databricks Workspace Admin Python Databricks SQL Databricks Workflows and PySpark. This role involves working in a hybrid model with day shifts and no travel requirements. The Sr. Developer will play a crucial role in developing and maintaining our data infrastructure ensuring efficient data processing and analysis.
Responsibilities
Develop and maintain scalable data pipelines using Spark in Scala and PySpark.
Oversee the administration of Databricks Workspace to ensure optimal performance and security.
Provide expertise in Python programming to develop and automate data processing tasks.
Utilize Databricks SQL to perform complex queries and data analysis.
Implement and manage Databricks Workflows to streamline data processing and integration.
Collaborate with data engineers and analysts to understand data requirements and deliver solutions.
Ensure data quality and integrity by implementing best practices and performing regular audits.
Optimize data processing performance by fine-tuning Spark and Databricks configurations.
Troubleshoot and resolve issues related to data pipelines and Databricks environment.
Document data processing workflows and maintain comprehensive technical documentation.
Stay updated with the latest advancements in data engineering and Databricks technologies.
Contribute to the continuous improvement of data infrastructure and processes.
Support the team in achieving project milestones and deliverables on time.
Qualifications
Possess strong expertise in Spark in Scala and PySpark for developing data pipelines.
Have extensive experience in administering Databricks Workspace for optimal performance.
Demonstrate proficiency in Python programming for data processing and automation.
Show advanced skills in Databricks SQL for complex data queries and analysis.
Have experience in implementing and managing Databricks Workflows.
Exhibit strong problem-solving skills and the ability to troubleshoot data pipeline issues.
Possess excellent communication skills to collaborate effectively with team members.
Have a keen eye for detail and a commitment to ensuring data quality and integrity.
Stay proactive in learning and adapting to new data engineering technologies.
Demonstrate the ability to work independently and manage multiple tasks efficiently.
Show a strong understanding of data processing best practices and optimization techniques.
Have a proven track record of delivering high-quality data solutions in a timely manner.
Exhibit a passion for leveraging data to drive business insights and impact.
Certifications Required
Databricks Certified Data Engineer Associate Apache Spark Developer Certification