Skip to main content

Topic Analysis to Enhance Automated Resolution Rates

note

This feature will be released soon. More details will be shared as changes are made. Stay tuned!

In customer service automation, understanding and leveraging conversation data is pivotal. One of the most powerful tools at your disposal is the analysis of topics automatically generated by your bot. This analysis helps identify opportunities to improve your automated resolution rate, leading to a more efficient and satisfying customer experience.

When your bot generates topics, it categorizes conversations based on the themes discussed. These topics are crucial for identifying areas where your bot can improve its performance. The default sorting of topics by Contained Resolution (CR) opportunity highlights the most significant opportunities for enhancing your bot's automated resolution rate.

To access topics page:

  1. Log in to the yellow.ai platform.
  2. Open Analyse > Conversation logs.
  3. Navigate to Topics.

image

Key metrics for topic analysis​

CR opportunity​

Contained resolution(CR) opportunity metric represents the total opportunity a topic has to improve your overall automated resolution rate. It is calculated as:

{CR Opportunity} = {Unresolved conversations under a topic}/{Unresolved conversations across all topics}

This helps identify which topics have the most potential for improvement.

CR opportunity suggestion​

The star next to the Conversation share means that the Topic has knowledge base article suggestion generated by AI to achieve CR. image

Conversation share​

This metric shows the proportion of conversations involving a particular topic compared to all conversations:

{Conversation Share} = {Conversations under a topic}/{Conversations across all topics}

It helps prioritize topics based on their frequency.

For example, in the below screenshot, Out of 4021 conversations taken place in this bot, 144 belong to this topic. image

CR rate​

The contained resolution(CR) rate indicates the percentage of conversations on a topic that were successfully resolved by the bot without human intervention:

{CR Rate} = {Conversations in this topic that were contained AND resolved}/{Conversations in this topic}

A higher CR Rate signifies better bot performance in resolving issues autonomously.

Containment rate​

This metric measures the percentage of conversations on a topic that were not escalated to a human agent:

{Containment Rate} = {Conversations in this topic not handed over to a human agent}/{Conversations in this topic}

A higher containment rate indicates greater efficiency in handling the topic without needing human support.

Sentiment​

This metric assesses the sentiment of users during conversations about a specific topic. It shows the percentage of positive, negative and neutral conversations that have taken place while discussing about this topic. Understanding user sentiment helps in identifying areas where the bot's responses might need improvement to enhance customer satisfaction.

For example, in the below screenshot, out of 1045 conversations in this topic, 550 (52.6%) were positive, 364 (34.8%) were negative, and 131 (12.5%) were neutral.

image

Expanded insights into each topic​

Topic details​

Click on the Topic name to view a detailed analysis of the selected topic.

Topic details

Date filter​

By default, analytics for the selected topic are filtered to the past 30 days. You can adjust the time period by selecting a different date range.

Date filter

CR opportunity​

This represents the percentage of conversations that either were not resolved or not contained, calculated as:

CR opportunity = (Total unresolved/uncontained conversations in this topic) / (Total unresolved/uncontained conversations across all topics).

Conversation share​

This shows the percentage of total conversations for the selected topic relative to all conversations across topics, calculated as:

Conversation share = (Total conversations in this topic) / (Total conversations across all topics).

You can view all conversations under a specific topic by clicking View conversations.

View conversations

On the Conversations page, you can read each conversation belonging to that topic. Use the filter icon to refine conversations based on your criteria:

  • Contained resolution: Setting this to True filters conversations that are both contained and resolved. Setting it to False filters conversations that are either contained but unresolved, not contained but resolved, or not contained and unresolved.
  • Contained: Filters all contained conversations, regardless of resolution status.
  • Resolved: Filters all resolved conversations, whether contained or not.
  • User sentiment: Filter conversations by sentiment categories such as Positive, Negative, or Neutral.
  • Automation: Select Available to filter conversations whose resolutions are used for automation purposes.

Conversation filters

For each conversation, you will see details such as:

  • Analysis: Whether it was contained/uncontained
  • User query: The specific query
  • Resolution: Whether it was resolved/unresolved
  • User sentiment: Categorized as Positive, Negative, or Neutral

Conversation details

Visualized results​

Key metrics such as CR rate, containment rate, and user sentiment for the selected topic are displayed as graphs for the selected time period. These values, already available on the Topics page, are visualized in graph form to help you understand trends more effectively. You can view conversations pre-filtered by clicking View all.

Visualized results

CR opportunity suggestions​

The bot also provides suggestions based on analyzed conversations:

  • Improve Knowledge Base: Click View conversation to see conversations that the AI used to recommend a new knowledge base article.
  • Analyze conversations: Click View conversation to identify opportunities for bot improvement.

CR suggestions

Utilize topics for bot improvement​

By closely monitoring these metrics, you can gain actionable insights into your bot's performance and identify areas for enhancement. Here are some steps to leverage topic analysis effectively:

  • Prioritize high-opportunity topics: Focus on topics with high CR Opportunity to make the most significant impact on your automated resolution rate. These are the areas where improving the bot's responses can yield the highest returns.
  • Analyze low containment rate topics: Investigate topics with low containment rates to understand why users are being escalated to human agents. This can help in refining the bot's responses or providing better training data.
  • Enhance CR Rate: For topics with lower CR Rates, consider revising the bot’s dialogue scripts, adding more detailed FAQs, or improving the bot’s understanding through advanced natural language processing (NLP) techniques.
  • Monitor sentiment: Keep an eye on user sentiment for each topic. If users consistently express negative sentiments, it’s a signal that the bot’s handling of that topic needs improvement.
  • Iterate and test: Regularly update and test the bot’s responses based on the insights gained from topic analysis. Continuous iteration helps in gradually enhancing the bot’s performance and increasing the automated resolution rate.

Containment rate for 3rd party Inbox apps​

This feature is under development.

Containment rate refers to the number of conversations handled solely by the bot without agent intervention. There are two types of such chat transfers:

  1. Active handover

    • In Yellow Inbox, chat is transferred to an agent with the event agent-transfer.
    • 3rd Party Inbox: Bot is paused, and chat is handed over to an agent with the event configured at the bot-level.

  2. Passive handover

    • Schedule callback: Bot schedules a callback by collecting user details.
    • Email: Bot creates an email ticket.
    • Ticket in 3rd Party System/Trigger Event: Bot creates a support ticket in another system or triggers a custom event.

Current limitation​

Yellow Inbox transfer are tracked with the agent-transfer event. But while using 3rd Party Inbox, the transfer depends on bot-level configuration but that is not currently trackable.

Suggested solution​

With Bot Developer access, you can configure the Chat Handoff node. This node is typically connected to the node used to raise a ticket in a third-party system. When this node is encountered, the agent-session event is tracked in the User Engagement Session Table. This setup enables tracking of the number of sessions that end with a passive handoff to third-party ticketing systems.

chat handoff

image

In the example below, a successful action after raising a ticket in Zendesk connects to the Chat Handoff node. When a passive or offline ticket is created in Zendesk, agents respond via email, call, or other channels. Meanwhile, this handoff is tracked. The Chat Handoff node acts as an identifier for agent transfers, enabling you to track the containment rate effectively.

example