Usage alone won’t tell you. Check a sample of answers, see whether handoffs reached a person, notice which questions still come to HR, and count the effort of keeping it current.
Isn’t high usage a good sign?
It shows people tried it, not that it helped. A chatbot can be used often and still give answers people have to check with HR.
1. Sample the answers.
Pick a set of recent questions and check each answer: the right source, the current version, and whether it applies to the person who asked.
Practical check: Use the same checks every time, so one month compares with the next.
2. Follow the handoffs.
When the chatbot passed a question to a person, did it arrive? Did it carry enough detail? Did the employee get an answer?
3. Watch what still comes to HR.
If the same questions keep arriving by email, the chatbot may be missing a source, giving answers people don’t trust, or hard to find.
4. Count the upkeep.
Include updating sources, retesting after changes and correcting wrong answers. That effort belongs in the picture as much as the questions it answers.
5. Decide on a schedule.
Agree when you’ll review it, and what would lead you to keep it, change it, narrow it or stop.
A fictional monthly check.
- Answers sampled
- 20
- Correct and sourced
- 17
- Partly right
- 2, both missing an exception for part-time staff
- Wrong
- 1, from an outdated policy that has since been replaced
- Handed to HR
- 2, both arrived with the details needed
- Still coming to HR
- Parking questions, which the chatbot has no source for
- Upkeep
- 3 hours
Read more: Does AI Actually Save Time? How to Measure the Whole Job · Why an HR Chatbot Gives Bad Answers, and What to Check
The figures are fictional. This is general guidance, not advice for your situation.