NLU deals with training machines to read and converse in any human language. Making AI Models understand nuances of language is very complex problem. Thankfully, using Linguistic Semantics i.e. by creating a structured format of sentences, models are able to percieve natural language with good accuracy.
- Linguistic : study of language
- Semantics : study of the meaning of words, phrases or sentences i.e. how arranging group of words in particular fashion to derive meaning.
Question is - how does knowledge of word meanings or sentence formation help in better training AI Models?
Fortunately, solution is easy to gauge by learning about a few key concepts - Intents, Entities and Context.
Any sentence can be broken down into smaller components -
Intents - These are the literal meanings or core objective of any sentence like in the above example it is booking a flight.
Entities - These are facts or additional information that adds meaning to sentence, for example Delhi, New York and 11th August.
Context - In a day-to-day conversation, this generally refers to the underlying meaning of previous few exchanges. For Example - if a person is asking repeated questions about a product and says "Buy IT" , the IT here refers to the product.
Once you have designed Intents for your bot, it is essential to train them . Before you click on that 'Train' button on top, let's make sure you are aware of the best practices!
|DONTs ❌||DOs ✅|
|Do not add utterances in which the only variation is Upper Case/Lower Case||Do add at least 15-20 utterances per journey|
|Do not add utterances in which the only variation is Name, Date, City etc||Do ensure that there are equal number of utterances in each journey|
|Do not create multiple journeys which have similar purpose||Do merge journeys that are subsets of other journeys|
|Do not overfit the model while training||Do use the didYouMean (suggestions) feature extensively|
|Do not add utterances if a journey will only be triggered through 'Trigger Journey'||Do minimize False Positives|
|Do add abbreviations/shortforms in the “synonyms” section|
|Do not add single words as utterances||Add complete sentences|
This will make the model overfit and not learn the underlying sentence structure resulting in bad performance.
A few utterances (2-3) like the ones mentioned below are ok but ensure that there are other utterances that show the different variations in sentence structure
- apply for leave tomorrow
- apply for leave on 3rd
- The minimum number of utterances in each journey heavily depends on the complexity of the bot (number, type of journeys and quality of the utterances)
- More utterances are always better especially when there are less than 10 journeys.
- Try to maintain a balance in the number of utterances per journey
- The NLP model is robust enough to handle small variations in the number of utterances (difference of 3-5 utterances)
- For smaller bots (< 10 journeys) maintaining balance is important to ensure good performance.
- Having multiple journeys which have similar utterances will confuse the model since there is a high amount of overlap.
- Merge all these journeys into one single journey
In the example above apply-for-home-loan is a subset or part of the apply-for-loan journey. This means that apply-for-home-loan will have utterances that are very similar to apply-for-loan- Eg: “can you please help me apply for home loan?” , “Can you please apply for loan?”This will confuse the model during training
There are 2 Steps to fix this:
* Another option is to setup entities (eg: type-of-loan - Personal, Home can be a type of entity) within the journey.
- Move all utterances to the parent journey in this case apply-for-loan
- Create a step asking the user for additional details (in this case type of loan)
Eg: Feedback Journey
For these journeys do NOT add any user expressions/utterances Adding utterances here will unnecessarily increase the complexity of the NLP Model.
- When an incorrect journey is triggered with high confidence it is considered a False Positive.
- False Positives occur because of overfitting and spoil the customer/user experience
- These are minimized by following the best practices laid out in this document.
- If there are a lot of false positives during training (even after checking for overfitting) try raising the minConfidence threshold.
- The best strategy is to use didYouMean(Suggestions) feature and retrain the bot periodically with the new data.
- After ensuring that the model did not overfit (no False Positives) the next step is to enable the didYouMean feature
- When the user’s input is not recognized by the model the didYouMean feature elegantly handles the case as a fallback.
- This is especially useful in the first few weeks after deploying a bot in production when accuracy may be low.
- The self-learning capability allows the bot to improve the confidence of different types of user expressions.
- For short forms and abbreviations add all the possible variations in the Synonyms section located under “entities”
- The NLP pipeline will check for these abbreviations and replace it with the “full form” before passing it into the ML model which will increase the accuracy
📌 Training Checklist
- 15-20 Utterances in each journey
- Utterances in journeys are varied and unique
- Journeys are distinct and conflicting journeys have been merged
- Model does not overfit (False Positives have been handled)
- didYouMean (Suggestions) feature is enabled
- Mininum 2 intents are required to train a bot
Confidence score (0 to 1) is a rating generated for every utterance to show how accurate is that utterance in pointing towards the given intent.
- Once you have trained your bot. Every utterance under the NLU section will generate this confidence. A value between 0 to 1 (1 being good and 0 being bad) will be assigned to every utterance.
- Any utterance with lower value (lower confidence) will mean that the utterance is bad or does not go with the utterances added in that particular intent.
Now you can also run Generate utterance reports on your bot to evaluate how well your bot utterances are designed. The report will tell you how 'similar' your utterances within an intent and if there are any intents common in between journeys'
Request folks to generate utterance report after initial NLU setup and regularly at least once a month.
There is an option in the Studio/ Flows section called Tools, also available in the NLU section.
Under tools, you'll see the 'Test your bot' box where you can test your utterances and see the score and intent they are triggering and many more details.