Our client is a global leader in digital transformation and data-driven innovation, currently expanding their global data team to support a wide range of digital initiatives.
They are now seeking a Data Engineer to join their Data Service team and play a key role in building scalable data pipelines and analytics solutions.
Key Responsibilities:
Design, build, and optimize scalable end-to-end data pipelines for structured and unstructured data from diverse sources (APIs, databases, files).
Develop and manage robust ETL/ELT workflows for both batch and real-time data processing, ensuring high data quality and reliability.
Monitor and optimize pipeline performance using autoscaling, caching, and cost-effective strategies.
Leverage Databricks and Apache Spark (PySpark, Spark SQL) for distributed data processing and transformation.
Implement data storage solutions using Medallion Architecture and Delta Lake, ensuring schema evolution and ACID compliance.
Collaborate with data scientists, analysts, engineers, and business stakeholders to align data solutions with business goals.
Stay current with emerging technologies in data engineering, especially within the Azure and Databricks ecosystem.
Key Requirements:
Bachelor's degree in Computer Science, Information Systems, or a related field.
5-10 years of hands-on experience in data engineering, with a strong track record in building and optimizing data pipelines.
Proven expertise in Databricks, Apache Spark (PySpark, Spark SQL), Python, and SQL.
Experience integrating Databricks with Azure services (Data Lake Storage, Data Factory, SQL Database, Event Hubs).
Familiarity with both batch and real-time data processing tools such as Azure Event Hubs and Kafka.
Proficient in Git and Azure DevOps for CI/CD and source control, including Databricks Asset Bundles.
Strong understanding of data quality, monitoring, and performance optimization.
Excellent documentation, collaboration, and communication skills.
Mature, independent, and customer-oriented with strong time management and problem-solving abilities.
Our client offers competitive compensation and benefits, as well as a dynamic and fast-paced work environment.
Interested parties, please send your CV to Ron Lee on Ron.Lee@connectedgroup.com or contact on +852 39725817 for more details. For other available opportunities, please visit www.connectedgroup.com