The AI Assistant is available as a separate, licensed product for VisualSP customers and once properly licensed a member of the VisualSP Product team will enable it as a Supported Main Panel Application in the customer’s subscription.
When creating and customizing prompts, you learn a lot from how (and how much) they are being used. Real-world usage information helps you make improvements. For this reason the VisualSP Prompt Editor tool provides you with a ‘Usage’ tab that shows you details of how the current prompt is performing with your end users. Are users finding and clicking on this prompt? Maybe it needs more visibility using the Spotlight or Featured options. Are you seeing a couple of common followup questions? Adjust your prompt to make it more likely to provide that commonly needed information on the first click.
Analytics for a specific prompt can be viewed by opening the prompt in the Prompt Editor panel and clicking the 1) Usage tab in the left menu.
The Usage panel has the following features:
2) Direct Submissions – the number of times this prompt was used as the initial prompt in a conversation with the AI Assistant
3) Total Follow-Up Submissions – the number of follow-up submissions that followed the direct submission of this prompt
4) Unique Users – the number of unique users who directly submitted this prompt
5) Rating – the average rating that this prompt received from users
6) Rating Distribution – the count of each rating
7) Favorites – the number of times this prompt has been added to a user’s favorites
8) Tokens – count of the tokens associated with the submissions of this prompt in the current application scope
9) Applications – an application scope picker to view this prompt’s usage across different application scopes.
In AI, a “token” is the smallest unit of data that an AI model processes, typically representing a piece of text like a word, character, or even a sub-word, which is used as a building block for understanding and analyzing language; essentially, it’s how text is broken down before being fed into an AI model for processing, allowing the model to identify patterns and relationships within the data. If you’re expecting short responses but seeing large token counts for the output then it might indicate an issue with your prompt instructions.
Graph Control Buttons
10) Total Prompt Submissions
11) Total Prompt Tokens
12) Average Response Duration
Export Controls
13) Export Prompt Data as CSV
Usage may be filtered using the available Start and End Date as well as the Range field to measure a prompt’s usage over time.
Additional Usage metadata is shown for Submissions by User and Prompt Follow-Up Submissions to drill down into 3) and 4) above.
Submissions by User – includes the following:
14) User Email Address
15) Last Date of Submission
16) Count of Follow-Up prompts
17) Total Direction prompt submissions by the user
If the subscription setting for Display Anonymous User Identifiers is set to “Yes”, the user’s email won’t be shown and will be replaced with a random GUID that marks the prompt submission.
Prompt Follow-Up Submissions – includes the following:
18) Prompt text
19) User Email Address
20) Date of the follow-up
21) Submissions count
22) Conversation Depth – is a relationship between the follow-up prompt and the direct prompt in ascending order. The lower the depth, the closer the follow-up prompt is to the initial (direct) prompt. This may be relevant in refining the current prompt instructions to reduce the number of follow-ups required to achieve the desired response.
If the subscription setting for Capture Suggested Prompts or Capture User Prompts is set to “No”, the prompt text won’t be shown in the Follow-Up Submissions table.
A key reason for the follow-up submissions is that it can help editors further refine the instructions to the prompt to either add or remove information that is or is not relevant based on how people are following up. If there are many similar follow-up prompts, then it may be helpful to include the details necessary in the instructions to help answer the question more directly rather than relying on the follow-up. Additionally, this may also spark ideas for new prompts.