Every AI agent demo works. It is run on a curated knowledge base, with predictable questions, and nobody impatient on the other side. Your operation has none of those three things.
The questions that reveal the most about an AI agent vendor are about failure, not capability: what happens when the answer is not in the base, when the visitor gets annoyed, when the model goes down, when you want to leave. Every demo shows capability; only real operation shows behaviour under failure.
About the answer
- What does the agent do when the answer is not in the base? The right answer is admit it and escalate, not improvise.
- How do you stop it asserting something the content does not support? Ask to see the check, not the promise.
- Can I see which passages produced each answer? Without that, debugging a wrong answer is guesswork.
About the hand-off
- What triggers a hand-off? Explicit request, repeated failure and detected frustration should all be on the list.
- What travels with the conversation when it reaches a person? If the transcript does not, the customer repeats everything: the single largest source of irritation in an automated channel.
- Can I declare topics that are never automated? Refunds, complaints and anything with legal weight usually belong there.
About latency and voice
- What is the median latency to the first token, rather than to the complete answer?
- In voice, what happens when both people speak at once? It is the fastest test of whether it is genuinely streaming.
About data
- What is masked before text reaches the model and the logs? See PII masking.
- Where is the data stored, and for how long?
- If I leave, what do I take? Transcripts, leads and the knowledge base should be exportable without depending on goodwill.
About integration
- Is there a signed webhook for the events that matter? It is what guarantees you can connect whatever the integration list does not cover.
The question that closes it
After the twelve, one remains: “what does your product not do well?”. Someone who answers with a concrete limitation is describing a real product. Someone who answers that it does everything well is describing a presentation. Our comparisons were written to that standard, each one includes the scenario where the alternative still wins.
