Test and deploy your agent

Running Simulations and reading Quality reports

Running simulations and reading quality reports are essential steps to ensure your AI customer agent delivers accurate, consistent, and reliable answers. These tools help you validate the agent’s performance, identify knowledge gaps, and maintain high-quality responses over time. Understanding how to use both effectively will empower you to optimize your AI and confidently deploy it in customer-facing scenarios.

running simulations to validate AI behavior

Simulations are scripted conversations designed to test your Customer Agent automatically. They simulate real customer interactions by running through predefined question-and-answer sequences, including follow-ups and procedure clicks. This automated testing helps catch regressions and validate edge cases before any changes go live, making improvements to your Living Knowledge safe and reliable.

key features of simulations

• Scripted turns: These are the customer’s messages written as a sequence, guiding the agent through specific scenarios.
• Conditions: Simulations run under defined contexts such as audience segments, topics, and surfaces, mirroring real deployment settings.
• Expected outcomes: You specify what the agent should say or do, allowing the system to score each run based on correctness.
• Suites: You can group multiple simulations into suites that run together on a schedule or on demand, such as daily regression checks or pre-release validations.

when and how to use simulations

Simulations are particularly useful:

• Before making any changes to Living Knowledge that might affect answers.
• Prior to publishing new components or Moments.
• When scaling your AI to new markets or languages.

Until simulations are fully available, you can rely on Preview for ad-hoc checks, Quality reports for scoring against control questions, and monitoring live Conversations to spot issues.

understanding and using quality reports

Quality reports provide a quantitative measure of your AI’s answer quality by running a curated set of control questions against the agent and scoring each answer. These reports give you consistent metrics over time, making it easier to track improvements or regressions.

key concepts in quality reports

• Control questions: These are representative questions you expect customers to ask, paired with your desired answers.
• Control answers: The benchmark answers you set for each control question.
• Report runs: Each time you generate a report, the agent answers all control questions, and scores are calculated by comparing generated answers to control answers.
• Report score: The average score across all questions, providing a quick health indicator.
• Per-question score: Detailed scores for each question, highlighting specific areas needing attention.

how to add control questions and generate reports

You can add control questions manually or pull them from your top real customer questions found in the Analyze > Performance section. Adding popular questions ensures your quality checks focus on what matters most to your users.

Once you have a set of control questions and answers, generate a quality report from the Test > Quality reports > Reports tab. After the report runs, you can open it to review each question’s score and compare the agent’s answer side by side with your control answer.

interpreting report results and improving scores

Low scores indicate significant disagreement between the agent’s answer and your control answer. By examining the sources the agent used, you can identify whether the underlying content needs updating or if the control answer should be revised.

Improvement steps include:

• Fixing or updating the source documents feeding Living Knowledge.
• Adding new sources with accurate information.
• Adjusting control answers if the agent’s response is actually correct.

when to run quality reports

• After any change to Living Knowledge to verify that answer quality is maintained or improved.
• On a weekly basis as a regular health check.
• Before presenting AI performance to leadership or auditors.

integrating simulations and quality reports into your workflow

Together, simulations and quality reports form a robust testing and monitoring framework. Simulations proactively validate complex conversational flows and edge cases, while quality reports continuously measure answer accuracy on key questions. Using both tools regularly helps you maintain a high-performing AI agent that meets customer expectations and adapts safely to changes.

conclusion

Running simulations and reading quality reports are critical practices for managing your AI customer agent’s performance. Simulations automate scenario testing to prevent regressions, while quality reports provide measurable insights into answer accuracy. By incorporating these tools into your regular workflow, you ensure your AI remains reliable, up to date, and aligned with your business knowledge.