Job Description

Edison Smart is supporting a confidential search for a Senior Evaluation & Safety Engineer to help build reliable, safe and production-ready Generative AI capabilities within a large organisation.

The role will shape how GenAI applications are tested, monitored and governed, working across AI engineering, data, security, risk and business teams. You’ll develop the evaluation frameworks and safety controls needed to move AI solutions from experimentation into dependable use.

What you’ll do

• Design and build evaluation frameworks for LLM and RAG applications, covering accuracy, relevance, groundedness, completeness and task success.

• Create benchmarks, test datasets and automated evaluation pipelines to compare models, prompts and system versions.

• Develop safety controls and test for hallucinations, prompt injection, data leakage, harmful outputs and policy violations.

• Integrate evaluation and safety checks into the GenAI development and release lifecycle.

• Monitor deployed solutions, investigate issues and use findings to improve model and system performance.

• Build clear reporting and observability for quality, risk, latency and cost.

• Partner with engineering, data, cybersecurity, risk, compliance and business teams to translate requirements into practical tests and controls.

• Support responsible AI practices, including human review and escalation for higher-risk use cases.

What you’ll bring

• Strong software engineering experience, including hands-on Python development.

• Experience building and evaluating production LLM or RAG applications.

• Knowledge of evaluation methods for generative AI, including automated metrics and human assessment.

• Experience with LLM orchestration frameworks, APIs and cloud-based AI services.

• Understanding of safety testing, guardrails, red teaming, prompt injection and responsible AI controls.

• Ability to build evaluation tooling, integrate it into deployment workflows and communicate results clearly.

• Experience working with sensitive data or in a regulated environment is highly valued.

• Strong communication skills and the ability to work across technical and non-technical teams.

This is an opportunity to help establish the standards and engineering practices that make GenAI dependable at scale.