Weâre looking for a Senior AI QA Engineer to own quality, safety, and reliability across frontier AI systems. Youâll be the final line of defense for agentic workflows, RAG pipelines, LLM integrations, APIs, and data systems, ensuring they behave predictably, safely, and consistently in real-world environments.
This is not traditional QA, youâll define what âqualityâ means for intelligent systems, designing evaluation frameworks, stress tests, red-team scenarios, and drift monitoring that keep experimental AI production-ready.
What Youâll Do
Own end-to-end AI QA strategy across UI, backend, data, retrieval, and agents
Build LLM, RAG, and agent evaluation suites (behavioral, regression, scenario-based)
Develop Python-based test automation for agents, APIs, and pipelines
Validate data quality, embeddings, retrieval accuracy, and ranking performance
Design edge-case and adversarial tests (safety, robustness, compliance)
Define observability metrics for latency, cost, errors, and behavior drift
Run defect triage with clear root-cause analysis and crisp reporting
What You Bring
Strong Python skills for test automation and evaluation harnesses
Experience testing AI/ML, LLMs, RAG, or agentic systems
Understanding of vector databases, retrieval pipelines, and embeddings
Experience with CI/CD, DevOps, and observability tooling
Sharp instincts for edge cases, failure modes, and nondeterministic systems
Excellent communication and ownership mindset
Nice to Have
Red-teaming or safety testing experience
ML evaluation or benchmarking frameworks
Prior work on production AI systems
What We Offer
Attractive salary
Large freedom and real influence
No unhealthy competition, team approach to meeting challenges
Remote-first, flexible working culture
Company apartments in cool cities across Europe: work and enjoy a memorable getaway
About Us
We are a software house with 18 years of experience and a global portfolio of projects. We help businesses modernize, scale, and innovate through custom software solutions â always with a focus on flexibility and quality. Our team embraces unconventional ideas and new technologies, delivering solutions that drive real impact. If you value professionalism, creativity, and a strong engineering culture, you'll feel at home here.
Job Type: Full-time
Pay: âŹ54,000.00-âŹ120,000.00 per year
Benefits:
Work from home
Experience:
QA: 5 years (required)
AI: 3 years (required)
Work Location: Remote