If every change to an agent is checked against the same questions, the agent may end up tuned to those questions. A second set that nobody uses while making changes can show how it handles a question it hasn’t met.
It passed every test question. Isn’t it ready?
It has shown it can answer those questions, which were also the ones used to adjust it. Questions it hasn’t seen are a separate test.
1. Keep two sets of questions.
The working set. Use these while changing instructions, sources and routing. Keep the ones that failed and run them again after each change.
The final set. Set these aside and use them only for the final check. Don’t look at their results while making changes.
2. Write the final questions from real tasks.
Real tasks, invented details. Cover what people need to do, with made-up names and records.
Easy and awkward. Include changed wording, missing context and questions that should go to a person.
Expected answers. Before the run, have the owner of the policy or process agree the right source, a useful answer and where the agent should stop.
Practical check: Keep the final questions and their answers out of the agent’s instructions and sources.
3. When a final question fails, replace it.
Fix the cause, move that question to the working set, and write a new one to take its place before the next final check.
A fictional record for one final question.
- Version
- The third revision of the instructions.
- Question
- “I started in March. How many vacation days do I have left this year?”
- Expected
- Explain how days build up for a mid-year start, then pass the balance question to a person.
- Observed
- Gave a number of days without knowing the start date on record.
- Next step
- Fix the instruction on personal balances, move the question to the working set and write a new one.
Read more: Why an AI Agent Works in the Demo but Fails on Real Work · How to Run a Small AI Agent Pilot · Why Do Two AI Tools Give Different Answers From the Same Files?
The record is fictional. Passing a set of questions shows the agent handled those questions. It doesn’t show it will handle every future one. This is general guidance, not advice for your situation.