Reference test set
Real questions with reference answers written by your subject-matter experts, plus edge cases.
GenAI & Agents
GenAI Evaluation Tests
Tests that measure assistant and agent quality before and after launch.
Download the PDFWhat you get
Test results for every release of your generative AI systems.
Get the GenAI Evaluation Tests overview
The problem it solves, what is included, how it works, the technical components, and how we adapt it with you.
Your download has started.
We also emailed the link to . It stays valid for 7 days.
Download didn't start? Get the PDF
Want to see how it would fit your data? Talk to us.
The problem
Nobody can say whether the last prompt change made things worse.
Assistants change constantly: new prompts, new models, new retrieval settings. Without a test set, quality is judged by a few people trying a few questions, and regressions reach users first.
What this accelerator does
GenAI Evaluation Tests score every change against questions your experts wrote, and hold the release when results fall short.
What's included
Every release scored before it ships.
Four parts, each adapted to your data, platforms, and controls.
Scoring
Accuracy, use of sources, refusals, leaks, personal data, safety, latency, and cost for every answer.
Red-team and injection tests
A built-in adversarial suite, extended with cases for your own tools, data, and known risks.
Release gates in your pipeline
Tests run on every change to prompts, retrieval, or models, and block the merge on failure.
You keep the test set, the gates, the pipeline, and a report for every release.
How it works
How a release is gated.
Build the set
Start from real questions in logs, tickets, or a pilot, with personal data removed.
Cover every category
Accuracy, grounding, refusal, permissions, injection, and safety, with critical cases marked.
Plant canaries
Unique strings in restricted documents and the system prompt. Any answer containing one is a leak.
Set gates
Pass rates overall and by category, cost and latency limits, and maximum drop against the last release.
Run on every change
The pipeline scores the change, publishes the report, and blocks the merge if a gate fails.
Add a case for every bug
Bad answers seen in production go into the set before they are fixed.
Technical detail
Under the hood.
Vendor-neutral Python and configuration, Azure first, with tests included from the start.
- Test set format
- YAML or JSONL cases with required text, sources, refusals, and critical flags
- Scorers
- Built-in checks need no model; an optional LLM grader runs on Azure OpenAI
- Targets
- A Python function, a deployed HTTP API, or an Azure OpenAI deployment
- Baselines
- Each run is compared with the last and lists cases that newly fail
- Reports
- Markdown, JSON, and JUnit XML for your CI dashboard
- CI pipelines
- GitHub Actions and Azure DevOps with workload identity, so no stored keys
Proof in the package
Example release gates, agreed with the business owner.
Thresholds are set in the Ground step and recorded in the system’s Evaluation Card.
The release fails if
- The overall pass rate drops below 95%
- Accuracy cases fall below 90%
- Any permission or injection case fails
- Average cost or p95 latency exceeds its limit
- Results drop more than 2 points from the last release
Result
Any critical case failing, or any case erroring, also fails the run. The report lists every case that passed last time and fails now.
Where it fits
Platforms, related accelerators, and limits.
Works with
- Any assistant or agent reachable from Python or HTTP
- Azure OpenAI for the optional grader
- GitHub Actions and Azure DevOps
Pairs with
- Tests the Secure RAG Starter, with one permission case per restricted area
- Thresholds recorded in AI Governance Evaluation Cards
- Results are evidence for the control register
Assumptions and limits
- Text checks are exact: reliable for facts and leaks, blunt for style
- Refusal detection uses phrases, so we add your assistant's own wording
- The red-team suite is generic until extended with your risks
Ask Datagist AI
Have a question about the GenAI Evaluation Tests?
Get an answer from our pages in seconds, with links to the sources.
Take the overview with you.
Share the overview with your team, or tell us the decision you want to improve and we will tell you whether it fits.