Automatic evaluation of the agent's conversations
How XBASE scores the AI agent's conversations against its objectives: how to turn it on, when it runs, where to see the scores and how much it uses.
Updated on
Automatic evaluation reads your AI agent's conversations and gives each one a score from 0 to 100, based on the objectives you define. It shows what the agent did well, where it failed and what to change in the instructions. In XBASE, the feature is called Observability, works for Native agents and comes turned off.
How it works
- The evaluation runs a few minutes after the agent's last reply, not on every message. If the agent replies again before that, the wait starts over. This way, the conversation is read after it has settled.
- If the conversation continues later, it is evaluated again, including what happened afterwards.
- For each objective, the evaluation says whether the agent met it and gives a score. The conversation's score is the average of the objectives that applied.
- Along with the score come a summary, strengths, weaknesses and recommendations to improve the agent.
- The evaluation only reads the conversation: the customer sees nothing and no message is sent.
How to turn it on
The evaluation only runs with two things set up in the agent's Panel:
- Observability on. In the Observability & Analysis card, click Enable Observability. The same switch is in Advanced → Agent Settings → Observability.
- At least one objective. In the Agent Objectives card, click Add Objective, fill in the Title and Description and click Add.
Without an objective, nothing is evaluated, even with Observability on.
What objectives are
An objective is what you expect the agent to do in the conversation, written in plain language. Examples:
- "Ask for the name and email before giving the price";
- "If the customer wants to cancel, offer another option first".
Use the Description to say when the objective applies and what counts as met: the evaluation reads the title and the description.
Each objective gets a score from 0 to 100 and a result:
| Result | When |
|---|---|
| Met | the agent did what the objective asks |
| Partial | the agent did only part of it |
| Not met | the objective applied to the conversation and the agent did not meet it |
| Not applicable | the objective's situation did not happen (e.g. the customer never mentioned canceling) |
A Not applicable objective does not count toward the average, so it does not lower the conversation's score.
What counts toward the score
- Only the evaluated agent is scored. The team's messages are left out of the evaluation. What another agent replied earlier in the same conversation is only used as context.
- Current service. If the conversation was idle for more than 7 days, only what came after the pause counts. In very long conversations, the most recent messages are used.
- The agent must have replied. A conversation where the evaluated agent did not reply is not evaluated.
- Technical failures do not cost points. If the agent's reply did not go out because of a platform error, it does not count against the agent.
- Objectives are not retroactive. A new or edited objective is not applied to replies the agent gave before that.
Where to see the scores
On the agent's Observability screen
In the Panel, Observability & Analysis card, click View metrics and reports. The screen has three tabs:
- Performance: choose Month or Week and see the period's Overview, with Evaluated Conversations, Average Score (0 to 100), Success Rate (objectives met out of those that applied), Evaluations (counts re-evaluations), Problematic Conversations and Pending learnings.
- Problematic Conversations: conversations where some objective was Not met or scored below 60. Each one shows the Score, Mark as reviewed and View conversation. Filter by Pending, Reviewed or All.
- Learnings: suggestions the AI draws from the agent's conversations. Apply them to the instructions (Apply suggestion automatically), adjust them first (Edit Manually) or discard them. The agent's Learnings tab opens the same list.
Before the first evaluation, the screen shows Observability Enabled and has no numbers yet.
In each conversation, in Chatcenter
Open the conversation, click the three-dot menu (⋮) at the top and choose Conversation Score. The panel shows the score, the Evaluation Summary, the Score by Objective (with the analysis and evidence), the Strengths, the Weaknesses and the Recommendations. A conversation without an evaluation shows Score not available.
Token usage
Each evaluation uses an AI model and uses tokens, like an agent reply. The usage shows in Usage & consumption and is deducted from the workspace balance, like the replies.
Since each conversation is evaluated after the last reply, and again if it continues, usage grows with the number of the agent's conversations.
To avoid this usage:
- turn off the agent's Observability: nothing else is evaluated, not even the evaluations that were waiting;
- or remove all objectives: without an objective, the evaluation does not run.