Job Title: Data Engineer
Location:
Australia
Employment Type:
Full-time/Part-time
Reports To:
Data Engineering Manager / Chief Data Officer
Job Overview
We are looking for a highly skilled and detail-oriented
Data Engineer
to join our growing data team. As a Data Engineer, you will design, build, and maintain robust data pipelines and infrastructure to support the company's data-driven decision-making processes. You will work closely with data scientists, analysts, and other stakeholders to ensure seamless access to high-quality data, helping to turn complex data into actionable insights.
Key Responsibilities
Data Pipeline Development:
Design, develop, and maintain scalable and efficient data pipelines to move data from various sources (e.g., databases, APIs, flat files) into data storage systems (e.g., data lakes, data warehouses).
Data Integration:
Integrate and consolidate large datasets from different sources, ensuring data quality, consistency, and accuracy.
Data Transformation:
Implement data transformation processes, including cleaning, aggregating, and enriching data for analytics and reporting.
ETL Processes:
Design, implement, and manage ETL (Extract, Transform, Load) processes to ensure that data is processed and stored efficiently.
Database Management:
Design and manage relational and non-relational databases to support data storage and retrieval. Experience with cloud databases (e.g., AWS Redshift, Google BigQuery, Azure SQL) is a plus.
Collaboration:
Work closely with data scientists, data analysts, and business stakeholders to understand data requirements and provide the necessary infrastructure and tools for data analysis.
Optimization:
Continuously monitor and optimize data workflows, database performance, and data pipelines for speed and efficiency.
Data Governance:
Ensure proper data governance practices are followed, including data security, compliance, and documentation of data processes.
Automation:
Automate repetitive data processes and work towards improving the overall data engineering efficiency.
Cloud Solutions:
Leverage cloud platforms (AWS, Google Cloud, Azure) to manage and process data at scale.
Skills \& Qualifications
Experience:
Proven experience as a Data Engineer or in a similar role (2+ years preferred).
Programming Languages:
Strong proficiency in programming languages such as Python, Java, or Scala for building data pipelines and automation.
ETL Tools:
Experience with ETL frameworks and tools such as Apache Airflow, Talend, or similar.
Data Warehousing:
Familiarity with data warehousing concepts and tools (e.g., Amazon Redshift, Google BigQuery, Snowflake).
Database Management:
Proficiency in working with SQL and NoSQL databases (e.g., MySQL, PostgreSQL, MongoDB, Cassandra).
Data Pipeline Frameworks:
Experience with frameworks like Apache Spark, Apache Kafka, or Hadoop for large-scale data processing.
Cloud Platforms:
Hands-on experience with cloud platforms like AWS, Google Cloud, or Azure, and knowledge of cloud-native data services.
Big Data Technologies:
Familiarity with big data tools and technologies such as Hadoop, Spark, or similar.
Data Modeling:
Understanding of data modeling and normalization techniques for efficient storage and retrieval of large datasets.
Version Control:
Proficiency in version control systems such as Git.
Problem Solving:
Strong analytical and problem-solving skills with the ability to handle complex data issues.
Communication Skills:
Excellent communication skills with the ability to explain technical data concepts to non-technical stakeholders.
Desirable Skills
Experience with containerization tools (e.g., Docker, Kubernetes) for deploying data pipelines.
Familiarity with real-time data processing frameworks (e.g., Apache Kafka, Apache Flink).
Exposure to machine learning pipelines and integration with data pipelines.
Knowledge of data governance and compliance (e.g., GDPR, HIPAA).
Familiarity with business intelligence tools such as Tableau or Power BI for data visualization.
Exposure to DevOps practices and CI/CD pipelines for data engineering.
Education \& Certifications
A degree in Computer Science, Data Engineering, Information Technology, or a related field is preferred.
Relevant certifications in cloud platforms (AWS Certified Data Analytics, Google Cloud Professional Data Engineer) are a plus.
Why Work With Us?
Competitive Salary \& Benefits:
We offer a competitive salary, health benefits, and paid time off.
Career Growth:
Opportunities for professional development and career progression within the data and technology teams.
Dynamic Work Environment:
Be part of a fast-paced and innovative company that values creativity and technical excellence.
Cutting-edge Technology:
Work with the latest technologies in the field of data engineering and cloud computing.
Work-Life Balance:
Flexible working hours and remote work options to support your personal life.
Collaborative Culture:
Join a team of passionate professionals dedicated to making a real impact through data.