A leading Data \& Analytics consultancy is looking for an AI Engineer (Databricks / Lakehouse AI) for their practice in Sydney
A brief job descroption is below with a more detailed one after that
Required
Strong proficiency in Python
Hands-on experience with Databricks
Solid understanding of machine learning concepts
Experience with Apache Spark
Familiarity with SQL
Experience building and deploying ML models in production
Exposure to LLMs, embeddings, vector search, and RAG architectures
Preferred
Experience with Azure
Experience with Mosaic AI, Databricks Feature Store, and Unity Catalog
Background in data engineering or analytics engineering
DETAILED SPECIFICATION
Role Overview
The AI Engineer is responsible for designing, building, and deploying AI-powered solutions on the Databricks Lakehouse platform. This role bridges data engineering, machine learning engineering, and applied AI, enabling scalable analytics, predictive models, and generative AI use cases across the enterprise.
The ideal candidate has hands-on experience with Databricks, strong Python and Spark skills, and practical exposure to machine learning and GenAI workloads in production environments.
Key Responsibilities
AI \& Machine Learning Development
Design, build, and deploy machine learning and AI models using Databricks Machine Learning and Mosaic AI
Develop end-to-end ML pipelines including data preparation, feature engineering, training, evaluation, and inference
Implement LLM-based solutions (e.g. RAG, fine-tuning, prompt engineering) using Databricks and open-source models
Integrate ML models into downstream applications via batch and real-time inference
Data Engineering \& Lakehouse Enablement
Build scalable data pipelines using Apache Spark, Delta Lake, and Databricks Workflows
Collaborate with data engineers to ensure high-quality, governed feature and training datasets
Leverage Unity Catalog for secure, governed access to data and AI assets
MLOps \& Productionisation
Implement MLOps best practices using MLflow for experiment tracking, model registry, and lifecycle management
Automate model deployment, monitoring, and retraining pipelines
Monitor model performance, data drift, and operational metrics in production
Collaboration \& Stakeholder Engagement
Work closely with data scientists, engineers, architects, and business stakeholders
Translate business problems into AI-driven solutions with measurable outcomes
Contribute to AI standards, reusable frameworks, and best practices within the organisation
Required Skills \& Experience
Core Technical Skills
Strong proficiency in Python for data science and AI workloads
Hands-on experience with Databricks (notebooks, jobs, MLflow, Delta Lake)
Solid understanding of machine learning concepts (supervised/unsupervised learning, model evaluation)
Experience with Apache Spark for large-scale data processing
Familiarity with SQL for data exploration and transformation
AI \& GenAI Experience
Experience building and deploying ML models in production
Exposure to LLMs, embeddings, vector search, and RAG architectures
Experience using frameworks such as Hugging Face, LangChain, or similar (preferred)
Cloud \& Platform
Experience on Azure (preferred), AWS, or GCP
Understanding of cloud security, identity, and cost management considerations for AI workloads
Preferred Qualifications
Experience with Mosaic AI, Databricks Feature Store, and Unity Catalog
Knowledge of CI/CD for ML pipelines
Background in data engineering or analytics engineering
Experience working in regulated or enterprise environments (e.g. Energy, Financial Services, Property)
What Success Looks Like
AI solutions are production-ready, scalable, and governed
Models deliver clear business value and are trusted by stakeholders
AI workloads are efficiently integrated into the Databricks Lakehouse
Best practices for MLOps, security, and cost optimisation are consistently applied
Why Join
Work on cutting-edge AI and GenAI solutions using the Databricks Lakehouse
Influence how AI is operationalised across the organisation
Collaborate with experienced data and analytics professionals
Continuous learning and exposure to emerging AI capabilities