A demo shows it can work on the examples chosen. Real requests bring missing details, the wrong source and errors between steps. Start with one failed case and check the input, the information it used and what happened at each step.
Will a better model fix it?
Maybe not. Find where it failed first. Keep the original request and the result it gave.
1. What did the person actually provide?
Look for missing identifiers, unclear wording, a pasted email thread or an unexpected file.
Compare. Was the demo input much cleaner or more complete?
Agree whether the process should ask for clarification, reject the input or carry on.
Practical check: Define what a usable input looks like, and what happens to the rest.
2. What information did it actually use?
Look at the document or record it retrieved, its date and its version.
Tell apart an answer that isn’t in the sources at all from one that is there but wasn’t found.
Check whether the answer is supported by what it actually used.
Practical check: A citation is something to verify, not a guarantee.
3. What happened between the steps?
Trace which tool or connection it used, and with what inputs.
Check whether each step succeeded, came back empty, failed or ended with no clear result.
Follow through. Where did the work stop? Who received the exception?
Practical check: Don’t infer completion from the final sentence.
4. Keep the failure as a test.
Build a small set of cases: known good ones, real failures and representative exceptions. Write down what should happen, what must not happen and how you’ll check. Rerun the same cases after every change, and repeat the important ones.
A record for one failure.
- Request and context
- What was asked, and which conditions mattered?
- Expected and actual
- What should have happened? What did?
- Change and retest
- What changed, who owns it and what did the checks show?
Then fix it, narrow it, replace it or stop, according to the evidence. Keep passwords and unnecessary personal details out of the record.
Read more: From Prototype to Production: What Changes? · How Do You Know an AI Agent Actually Finished the Task?
Passing a test set is evidence, not a guarantee. This is general guidance, not advice for your situation.