In this document, the following topics will be covered:
- What are intents?
- How to add intents and utterances?
- How to train Intents?
- How to test bots prediction-accuracy?
- How to resolve clashes in intents and utterance?
- Best practices to create intents and utterances
The words Flows and Journeys are used synonymously.
1. Intents overview#
NLU deals with training machines to read and converse in any human language. Making AI Models understand the nuances of language is a very complex problem. Using Linguistic Semantics i.e. by creating a structured format of sentences, models are able to perceive natural language with good accuracy.
- Linguistic : Study of language
- Semantics : Study of the meaning of words, phrases or sentences i.e. how to arrange groups of words in a particular fashion to derive meaning.
Knowledge of word meanings or sentence formation help in training the AI Models better with a few key concepts such as Intents, Entities and Context.
For example, any sentence can be broken down into smaller components -
|These are the literal meanings or core objectives of any sentence like in the above example it is booking a flight.||These are facts or additional information that adds meaning to sentences, for example, Delhi, New York and 11th August.||In a day-to-day conversation, this generally refers to the underlying meaning of the previous few exchanges. For Example - if a person is asking repeated questions about a product and says "Buy IT", the IT here refers to the product.|
On our platform, you can add your intents (what will be the intention of the bot users response) and utterances(what is the information that the bot user asks for when this intention is detected). Train the model to recognise such sentences.
Yellow.ai DynamicNLP is based on cutting-edge technology, Zero-shot Learning, that will allow you to bypass the tedious, complex, and error-prone process of model training. Continuous upgrade to the NLP engine helps our Dynamic AI Agents improve the intent performance, which ensures that your customers get accurate responses from day one. Our NLP has “seen” all different syntactic variations of sentences from billions of conversational data. With a better understanding of the context and intention of their queries, your customers enjoy a seamless experience. You can just add in your intents by following our Intent naming conventions and train the bot.
Even though Zero-shot model does not require manual addition of utterances, we will consider that scenario and discuss intent and utterances in this document.
Creating intents for Jimmy's cafe:
You are the owner of Jimmy's cafe and are setting up a bot for your website. Your customers would like to enjoy the simplicity of ordering a coffee through your bot. For this to happen, you want your bot to understand the different ways a customer may ask to place an order so that the flow you built gets triggered. I want coffee will be recognized as intent: #ordercoffee. I want a croissant will be recognized as intent: #ordersnack.
After creating the required flows, click on Intents in the Train dropdown. Use the
i key to access Intents directly from any Studio page.
2. Add intents and utterances#
To add intents and utterances, follow the given steps:
A prerequisite to training your bot with the required intents is to have a basic flow ready. Create your first flow in the studio module of the platform.
Click here to learn more.
After creating the required flows, click Studio > Train > Intents.
You can access Intents using the keyboard shortcut
There are several ways in which a customer would like to place an order for a cup of coffee:
- Place an order
- I want to place an order
- Place order, and many more...
You can group all these statements in an intent called order.
To add a new intent, click on +Add new Intent button and manually add the first Utterance.
With “Yellow.ai DynamicNLP”, NLP based on Zero-shot learning we eliminate the need for training the NLP model with utterances. To take advantage of this, follow the guidelines in Best Practices > 6.1 Intent Naming .
Add utterances to the intent. Utterances are phrases or queries that users may type in the bot conversation with an expectation of a response to that exact query.
There are two ways to add Utterances to an intent:
While adding utterances manually to your intent, you do not need to pay attention to the case of the utterance, the bot will consider all such scenarios.
Type in your utterance and click +Add to add the utterance to an intent.
Yellow.ai has data collected from over 100+ bots. This data is used to curate the suggested utterance section. In this section, you can see phrases similar to the first utterance you added.
The refresh button will allow you to access a fresh batch of utterances every time you click on it. You may add a few utterances by clicking the '+' sign next to them or add all of them by selecting the check box next to 'Suggested utterances'.
This would save you the effort of thinking of phrases and help you create intents in minutes.
3. Train intents#
Click Train Intents.
You can train the intent after adding 2 utterances, but it's recommended to train your intent after adding at least 15 utterances.
You can increase the number of epochs for training your intent. The number of epochs is set to 20 by default. However, they could be a deciding factor when it comes to underfitting or over-fitting the model.
After training intents you can connect it to your bot. You have to connect the flow you built to the intent 'Order'. To do this click the Start Trigger and configure the intent to the node.
Click the drop-down and select ‘#order’ intent.
Every time a user asks a query similar to the utterances within 'order' the flow you created would get triggered.
Congratulations! You trained and connected your first intent! 🎉🎉🎉
4. Test intents#
Once you have trained your intent you can test it for the results and retrain it (if required) based on the utterance report. There are two methods to test your intent.
To test your bot you can follow these steps:
- On the studio page, click the right panel to test your bot.
- To test your intent, type "Place an order" in your bot.
- You will see that your flow gets triggered.
To see what response is generated by the model when a user types a query. Click on Tools and the section 'Test your bot'.
You can test how confident your bot is about a phrase and whether it can identify the intent you just built.
As you can see in the above code, the model understands that the phrase is a part of the intent 'order' and is completely confident about it (0.999).
5. Resolve clashes in intents and utterances#
A bot is trained with multiple custom intents and entities to get the best result. There might be situations that will confuse the bot if the utterances are not classified correctly while training. These clashes that have resulted due to unclear utterance classification can be resolved by studying the utterance report.
Follow the given steps to learn:
- How to download an utterance report
- How to resolve clashes among utterances
- Open Studio > Train > Intents , click Generate utterance report.
Two reports will be sent to your registered email ID.
- Utterances within intents/faqs
- Utterances across intents/faqs
- Report for utterances within intents/faq highlights similar or extremely diverse utterances that must be edited within an intent/FAQ including clashes due to entity featurisation.
- Report for utterances across intents/faqs highlights similar utterances across the flows, it will recommend you to change any one utterance of the similar pair including clashes due to entity featurisation.
Utterance reports sent to your mail ID are used to evaluate how well your bot utterances are designed. You can learn how 'similar' your utterances are within an intent and if there are any intents common in between the flows.
It is recommended to generate an utterance report after initial Train setup and regularly at least once a month.
This report will point to the relationship between the two utterances as a conflict if they have a high similarity. It is a comparison between utterances of the same intents.
If the similarity is more than 50%, you must go to the respective Intents page and delete one of the similar utterances or rephrase the sentence.
If the similarity is less than 50%, it can be ignored.
This is a comparison between utterances of different intents.
If the similarity is more than 50%, you must go to any of the Intents page and delete similar utterances or rephrase the sentence.
6. Best practices#
This section is divided into:
- Best practices to follow while naming intents.
- Best practices to follow while adding utterances to the intents.
6.1 Intent naming#
There are guidelines for new bots and for the bots in productions. For your intent to work best globally, follow the respective guidelines:
- Intent names must be at least 3 words long with unique words and no special characters.
- Be mindful of intent names, make sure they are as descriptive as possible.
- Don't create intent names like intent test one, FAQ number one etc.
- Bad intent names will result in bad NLP performance (False positives) and unnecessary issues in the bot.
- For Cloud, it is possible to rename intent names.
- The more descriptive the intent name, the better (add names with more than 3 words).
- Avoid uncommonly and business-specific abbreviations: Example: PO (purchase order ), GMV, etc - use the full forms and add synonyms if necessary. Few common abbreviations like UPI, EMI, and HR are acceptable.
- Phrase the intent name as a verb followed by a noun. Example: get a premium receipt, pay renewal amount, fetch order status.
- Keywords and sentences less than 3 words will fallback to the existing bot model and will work as-is. These types of utterances will not go to the new model.
- This model is applicable and works well for FAQs as well (since FAQs are descriptive and longer sentences)
- Suggestions are enabled by default for all new bots - as this is critical for model improvement and to provide the full performance benefit.
Following are a few important pointers for bots that are already in production:
- Enable suggestions for bots where they may not be enabled. This ensures that the model is used to the fullest.
- Suggestions only show up for intents that are connected to the flows. Verify that unwanted flows are removed (or disconnected from intents).
- If the intent name is camelCase (eg: chatWithAgent) or has underscore/hyphens (eg: chat_with_agent, chat-with-agent), use the edit option to rename these following the guidelines mentioned in the above section(for new bots).
- Ensure that there is no Small Talk in FAQs / Flows. If these are present, delete them - platform small talk is enabled for all cloud bots.
- Enable suggestions for bots.
- To do this, in app.ym ensure that enableDidYouMean is set to true in app options in Function and in Tools → App Options → Prediction → Enable Suggestions.
- If there’s an existing DidYouMean function in default:response, remove it.
- Verify that the flow/journey DESCRIPTION is in line with the guidelines mentioned above.
- If these are not in line and are in camelCase or have special characters, change these by going to flow settings for that flow (you need not change the journey name, only the description can be changed).
- Ensure that there is no Small Talk in FAQs/Flows. If these are present, delete them and enable platform small talk in Context Management and enable Small Talk.
|DONTs ❌||DOs ✅|
|Do not add utterances in which the only variation is Upper Case/Lower Case||Do add at least 15-20 utterances per flow|
|Do not add utterances in which the only variation is Name, Date, City etc||Do ensure that there are an equal number of utterances in each flow|
|Do not create multiple flows which have a similar purpose||Do merge flows that are subsets of other flows|
|Do not overfit the model while training||Do use the didYouMean (suggestions) feature extensively|
|Do not add utterances if a flow will only be triggered through 'Trigger Journey'||Do minimize False Positives|
|Do add abbreviations/short forms in the “synonyms” section|
|Do not add single words as utterances||Add complete sentences|
This will make the model overfit and not learn the underlying sentence structure resulting in bad performance.
A few utterances (2-3) like the ones mentioned below are ok but ensure that there are other utterances that show the different variations in sentence structure
- apply for leave tomorrow
- apply for leave on 3rd
- The minimum number of utterances in each flow heavily depends on the complexity of the bot (number, type of flows and quality of the utterances)
- More utterances are always better especially when there are less than 10 flows.
- Try to maintain a balance in the number of utterances per flow
- The NLP model is robust enough to handle small variations in the number of utterances (difference of 3-5 utterances)
- For smaller bots (< 10 flows) maintaining balance is important to ensure good performance.
- Having multiple flows which have similar utterances will confuse the model since there is a high amount of overlap.
- Merge all these flows into one single flow.
In the example above apply-for-home-loan is a subset or part of the apply-for-loan flow. This means that apply-for-home-loan will have utterances that are very similar to apply-for-loan- Eg: “can you please help me apply for home loan?” , “Can you please apply for loan?”This will confuse the model during training
There are 2 Steps to fix this:
* Another option is to setup entities (eg: type-of-loan - Personal, Home can be a type of entity) within the flow.
- Move all utterances to the parent flow in this case apply-for-loan
- Create a step asking the user for additional details (in this case type of loan)
Eg: Feedback Flow
For these flows do NOT add any user expressions/utterances Adding utterances here will unnecessarily increase the complexity of the NLP Model.
- When an incorrect flow is triggered with high confidence it is considered a False Positive.
- False Positives occur because of overfitting and spoil the customer/user experience
- These are minimized by following the best practices laid out in this document.
- If there are a lot of false positives during training (even after checking for overfitting) try raising the minConfidence threshold.
- The best strategy is to use didYouMean(Suggestions) feature and retrain the bot periodically with the new data.
- After ensuring that the model did not overfit (no False Positives) the next step is to enable the didYouMean feature
- When the user’s input is not recognized by the model the didYouMean feature elegantly handles the case as a fallback.
- This is especially useful in the first few weeks after deploying a bot in production when accuracy may be low.
- The self-learning capability allows the bot to improve the confidence of different types of user expressions.
- For short forms and abbreviations add all the possible variations in the Synonyms section located under “entities”
- The NLP pipeline will check for these abbreviations and replace them with the “full form” before passing them into the ML model which will increase the accuracy
📌 Training Checklist
- Add 15-20 Utterances in each flow.
- Utterances in flows must be varied and unique.
- Flows are distinct and conflicting flows must be merged.
- Model should not overfit (False Positives have to be handled).
- When didYouMean (Suggestions) feature is enabled, make use of those suggestions.
- Minimum 2 intents are required to train a bot.