Test and deploy your agent
Running Simulations and reading Quality reports
Running Simulations and reading Quality reports
Keeping your Customer Agent accurate is not a one-time effort. As you add Sources, tweak Procedures, or adjust Personality, you need a way to confirm nothing broke. Two features in the Test section of the Unless platform handle this: Simulations and Quality reports. Simulations run scripted conversations automatically to catch regressions, while Quality reports score the agent against a fixed set of control questions. Together, they give you a safety net before changes go live and a measuring stick for ongoing performance.
Understanding Simulations
Simulations are scripted conversations that run against the Customer Agent automatically. They let you catch regressions and validate edge cases before a change goes live. This feature is in Beta and will be available soon.
The idea is simple: Preview is one human, one session - useful, but slow. Simulations turn the questions you care about into a regression suite that runs every time you change a Source, a Procedure, or a Personality. The suite is what makes Living Knowledge improvements safe.
Key concepts
- Scripted turns - The customer’s side of a conversation, written as a sequence of messages. Includes follow-ups and Procedure clicks.
- Conditions - The Audience, Living Context, topic, and surface the Simulation runs under. Same setup you would use in Preview.
- Expected outcomes - What you expect the agent to say or do. Used to score each run.
- Suites - Groups of Simulations that run together. Examples include a daily regression suite, a per-moment suite, or a pre-release suite.
What you can do
Once Simulations ship, you will be able to write scripted conversations that drive the agent through specific scenarios, group Simulations into suites and run them on a schedule or on demand, compare scores across runs to catch regressions, and trace failures to the Source, Procedure, or Living Context value that caused them.
When to use Simulations
Use Simulations before any change to Living Knowledge that could break existing answers, before publishing a new component or Moment, and before scaling to a new market or language.
What to do until Simulations ship
Three things cover most of what Simulations would do:
- Use Preview for ad-hoc checks
- Use Quality reports to score the agent against control questions
- Use Conversations to spot answers that need attention in production
Understanding Quality reports
Quality reports run a curated set of control questions against the agent and score each answer. They give you numbers with the same questions every time, so improvement is measurable. The quality control center helps you compare your desired answers to the answers provided by the AI for questions you have added.
Key concepts
- Control questions - Questions you would expect any customer to ask, with the answer you would expect the agent to give. Build the list once, run it many times.
- Control answers - The expected answers. Used as the benchmark when scoring.
- Report runs - Each time you generate a report, the agent runs every control question. Scores compare each answer to the control answer.
- Report score - Average of all question scores. Useful as a quick health indicator over time.
- Per-question score - The signal you actually act on. Low-scoring questions point straight at the Source that needs work.
Adding control questions
From the Most common questions in the last 30 days block in the AI insights page, click the plus icon next to a question to add it to the quality control center along with your desired answer.
You can optionally specify a segment that the question and answer belong to. A quick way to get started is by taking a look at the Most common questions in the last 30 days block in your AI zone insights. You can review the questions that get asked the most and click the icon with the + sign to add them to the quality control center.
Checking the answer quality for these popular questions is a good way to ensure that you have a solid base to work from. You can also ask your customer support team for suggestions as they are the experts when it comes to FAQs.
Running a report
Once you’ve added some control questions and answers, you can generate a report via the Reports tab. This is also where you will find a list of your reports over time.
Open “Test > Quality reports > Reports” and click “Run report”. The report appears in the list when it finishes.
Reviewing the report
You can then click open report to view the generated answers and see how they compare to your control answers. Each answer receives a score which makes it easier to review at a quick glance. This way you can focus on the answers that have a low score and explore why. There will also be an explanation of the score for each answer.
The first example highlighted is about whether Unless offers a free trial. While the correct answer is no, the generated answer is yes. This is likely because free trials were offered in the past and some mentions may have been left on the website. In this case, you can click the i icon for information, see the sources that were used to generate this answer, and update it. You can then re-run the report and see if the score improves.
The second highlighted example shows where the generated answer is correct but contains additional information when compared to the control answer. However, the extra information is still correct so the score remains high at a 7/10.
How scoring works
Each control question runs against the live agent. Scores compare the generated answer to the control answer using semantic similarity, not exact match. Small differences in wording get high scores. Real disagreements get low scores.
Improving low scores
A low score means the agent’s answer disagreed substantially with the control answer. Open the question to see the agent’s answer, the Sources it used, and the control answer side by side. To improve it, find the Source the agent used. Either fix that Source, add a new Source that has the right information, or update the control answer if the agent was actually correct.
When to run reports
Run reports once a week as a baseline, plus on demand after any major change. Use them after any change to Living Knowledge to confirm scores held or improved, and before showing the agent’s results to leadership or auditors.
Conclusion
Simulations and Quality reports serve different but complementary roles. Simulations will give you automated regression testing for scripted scenarios, while Quality reports give you a consistent, repeatable scorecard for the questions that matter most. Use both on a regular cadence, and you will catch problems before customers do. The key is consistency: build your control set once, run it often, and act on the low scores by fixing the underlying Sources.