Wrong-answer test · Dose suggestion
1 of 3 · The model · 8 runs, same input
Evidence pack · run manifest · independent of the model vendor and your build teamWorked example on a demo product
Free 45-minute session We map the ways your AI could be wrong, and what would catch it
Who is testing your AI?
We test what your product does when its AI is wrong.
Wrong-answer test · Dose suggestion
2 of 3 · The product · fed run 6 at the seam
Evidence pack · run manifest · independent of the model vendor and your build teamWorked example on a demo product
Wrong-answer test · Dose suggestion
3 of 3 · The evidence pack
Evidence pack · run manifest · independent of the model vendor and your build teamWorked example on a demo product
Why now
Four moments when testing your AI stops being optional.
If one of these happened in the last month, you have a risk nobody has tested. Forty-five minutes will tell you how big it is.
-
Something went wrong
Did a wrong answer reach a user?
Something let it through. We find the gap, then prove it stays closed on every run.
-
Someone is asking
Did a buyer ask how you test your AI?
NHS buyers, enterprise customers and investors now ask. Give them evidence, not a policy.
-
A change you did not make
Did your AI provider update a model?
Nothing in your code changed, but the answers did. We re-run your tests and show you every difference.
-
A change you made
Did you change a prompt last sprint?
One line of wording can change every answer. We test the change like the release it is.
We check three things.
Your AI will hallucinate, and its model will change on someone else’s schedule. These three checks decide whether that matters.
SAFEGUARDS
When the AI is wrong, does your product catch it?
We feed your product bad AI output on purpose and test the safeguards meant to stop it, one scenario at a time, on your staging environment, against a named risk.
Learn moreWhat we try to break.
ACCURACY
Are the answers actually right?
We check the answer too, but properly: only against reference answers your own experts define. Where a machine does the scoring, we check it against your people first, and report how closely it agrees.
Learn moreWhat usually goes wrong.
MODEL UPDATES
When the model is updated, what broke?
A fixed test set, a recorded baseline on today’s model, then the same set re-run after every change, with every difference reviewed by a person.
Learn moreWhat we try to break.
Getting started
From your AI feature to an evidence pack in six steps.
Step 1 is free. Steps 2 to 5 are a fixed 10-day first engagement. Step 6 repeats for as long as your product keeps changing.
-
Map the failure scenarios
A free 45-minute session. We agree the ways your AI could be wrong that matter most, and the safeguard meant to catch each one. You keep the map.
-
Build the test set
A fixed set of scenarios and cases, with your own experts’ definition of a wrong answer for each one.
-
Record the baseline
The set runs on today’s model, every case repeated, with everything about the run recorded so it can be reproduced.
-
Break it on purpose
At the seam where the AI’s answer enters your product, we feed it wrong answers and test whether each safeguard holds.
-
Hand over the evidence
Results by failure type, versions, dates and the name of the person who reviewed each one, in one reviewable pack.
-
Re-check on every change
New model, new provider or new prompt: the same set runs again and every difference is reviewed. This step repeats.
Who does the work
A named person, not a dashboard.
Run by Imran Ali
Twenty years testing software in regulated industries, where the record has to stand up. Imran runs the free session himself.
A named reviewer on every result
Every result, and every difference after a model update, is read and signed by a named person. Nothing is waved through by a machine.
Independent of model providers and tool vendors
We do not sell a model or an evaluation tool, and we are not owned by a company that does. The only thing we have to sell you is the testing.
Clear about what we do not do
No security penetration testing, no bias or accent testing, no monitoring of live answers, no making your model better, and no certificate. You decide whether to release. We give you the evidence to decide with.
-
20+
Years in regulated QA
-
10
Days, fixed first engagement
-
8
Runs per scenario
Every result has a name on it. You can see who we are before you speak to us.
More about us
Frequently asked
Questions we get asked before every engagement.
Do you test the AI model, or our product?
Your product. Model benchmarks tell you how a model performs in general; they tell you nothing about whether your product catches a bad answer before it reaches someone. We stand in for the model, feed your product answers that are wrong in specific ways, and test what happens next.
What happens when we change model or provider?
That is the question most teams have no answer for. We record a baseline on your current model, then re-run the same set after the change and review every difference. Nothing in your code changes and your tests still pass, but the answers can be different. The re-check is what closes that gap.
Who decides what counts as a wrong answer?
You do. Correctness is measured against reference answers your own experts define, or an expert reviewer agreed with you in advance. We never decide that your product’s answers are right, and we never claim correctness from invented data alone.
Do you need access to our systems, or real user data?
A staging environment and one engineer as a point of contact. No live user data. Anything that needs real records runs inside your environment, under your control, and we ask for written permission before any adversarial testing.
We already use an evaluation tool or service. Why would we need this?
Keep it. Evaluation tools and services score your AI’s answers. We test what your product does next: whether a wrong answer is caught, whether review can be skipped, what happens when the AI is down. The two work together.
We already have QA. Why would we need this?
Most QA is built for software that behaves the same way twice. These questions are different: how often does it hold, what changed when the model did, and does anything stop a wrong answer reaching someone. Teams with strong QA often still want an outside pair of eyes on those three.
Not sure what to test first? We’ll tell you.
Book a 45-minute failure-scenario session. We map the five ways your AI could be wrong that matter most for one feature, and the safeguard meant to catch each one. You keep the map, whatever you decide next.