How To Migrate Your Chatbot From IBM Watson Assistant To Rasa

How To Migrate Your Chatbot From IBM Watson Assistant To Rasa

Considerations When Moving Your Chatbot To Rasa From Watson Assistant

Introduction

IBM Watson Assistant has a very minimalist design & development interface, yet very efficient. The four chatbot pillars are well defined with a top layer of advanced functionality…

It will hardly ever be a seamless transition when you migrate from one chatbot platform to another. As there are fundamental design and data differences between environments.

Most chatbot platforms are constituted by the same four basic pillars:

  • Intents
  • Entities
  • State Machine / Dialog Management
  • Script
Image for post
Traditional Chatbot Architecture

There are of course variations between systems:

  • The extend to which the core and the NLU elements are separated.
  • The propriety structures and data types introduced to entities.
  • To which extend entities and intents are merged.
  • Special support features.

Watson Assistant Structure

The functionality available In the Watson Assistant console is clearly defined and intuitively structured. It is easy to learn and become familiar with the environment.

Image for post
Basic architectural structure of IBM Watson Assistant

Intents

IBM Watson Assistant (WA), at its core has a basic intent and entity structure. Intents are as minimalist as can be.

Image for post
Intents Listed

During the intent creation process, there are two features which aid in the defining of intents. Intent Recommendations and Intent Conflict Resolution.

Bot of these features translate into better defined intents, and translates nicely into the JSON export file.

Hence the leverage these functions lend to the intent creation process is not lost.

It must be noted that entities alone can be exported, in CSV format. But all data structures are lost and only the entity utterance examples are exported.

Entities

Entities within WA does not have any structure attached to them.

Image for post
Entities Listed

This simplifies the transition process.

Synonym Recommendations is a feature which improves the quality of the entities created.

Fortunately, this translates into the export of the skill.

Annotation is key in establishing truly contextual entities.

This is part and parcel of the export and translates well into Rasa.

Entities can also be exported in CSV format. But again, data structures are lost.

Dialog (Conversational Nodes)

This is where it becomes tricky. It must be noted, that IBM Watson does have a NLU API function within their cloud offering.

Image for post
WA web based dialog node and conversational state management.

But for WA, the chat environment is combined with the NLU portion.

This is a completely opposite approach compared to AWS Lex or Microsoft LUIS/Bot Framework.

The console is completely integrated with all of this functionality made available via an feature rich API.

The good news is that in the chatbot JSON export file creates a clear distinction between the intents/entities and the dialog flow.

Hence the manipulation of the file is easy as the segmentation of components are clearly defined.

Watson Specific Functionality

At the very top of the Basic architectural structure of IBM Watson Assistant diagram you will see a set of Watson specific features.

Image for post
Under Options you see Watson Assistant Specific Features

These are Webhooks, Disambiguation, Auto Learning, Auto Correction, Irrelevance Detection, System Entities & Fuzzy Matching.

These features really do augment the chatbot performance and drives down the user effort.

Unfortunately, on migration, these functions will be lost. But, it needs to be mentioned that these functions do not pollute or clutter the skill export. The export file is clean and readable. It is almost as if IBM separates skills into two segments.

The one is the user crafting the chatbot elements, and the Watson augmenting these elements.

Watson Assistant Export Compared To Rasa nlu.md Format

Approach

The general approach I will following migrating to Rasa from Watson Assistant would be to export the WA skill into a JSON formatted file.

From there you can create a conversion program to convert the JSON export into the Rasa nlu.md file format.

The lack of entity structures within WA bodes well for the migration process.

Intents

The format of intent exports in the JSON file from Watson Assistant:

{
"intents": [
{
"intent": "InformationLoan",
"examples": [
{
"text": "How can I get information on loans?"
},
{
"text": "How can I get more information on loans?"
},
{
"text": "how does applying for a loan work"
},
{
"text": "How does loans work"
},
{
"text": "Information on loans please"
},
{
"text": "where do I go for info on loans"
}
],
"description": ""
},

Compared to Rasa’s NLU Format:

## intent:ask_transfer_charge
- Will I be charged for transferring money
- do transfers cost something?
- is there a transfer charge?
- Is there a charge
- will i be charged for a transaction?
- do xfers cost something?
- is there a transfer fee
- is there a xfer fee
- how much is the transfer fee

Form this comparison creating intent training data from the WA export via text manipulation is straight forward.

Entities

I prefer chatbot development environments where the entities can be annotated within the intent. in many environments the intents and entities are too far removed; but here there is a convergence.

Below is an WA export example, with the intent name, with a text example. You will also see the entity name and the location of the entity.

{
"intent": "TransferMoney",
"examples": [
{
"text": "Can we transfer money from my savings account to my bond account",
"mentions": [
{
"entity": "Money",
"location": [
16,
21
]
},
{
"entity": "ToAccount",
"location": [
52,
56
]
},
{
"entity": "FromAccount",
"location": [
30,
37
]
}
]
},

Elsewhere the entity is defined in detail:

{
"entity": "Money",
"values": [
{
"type": "synonyms",
"value": "money",
"synonyms": []
},
{
"type": "synonyms",
"value": "R500",
"synonyms": [
"10000",
"400",
"R500"
]
}
],
"fuzzy_match": true
},

Comparing this to the nlu.md file format:

- I want to pay the [current balance](payment_amount) on my [credit card](account_type)

From the WA example the nlu.md format can be generated. The only impediment, within WA entities can be defined in isolation. Adding these to Rasa might proof problematic.

Dialog / Stories

Rasa’s approach to dialog management cannot be fore juxtaposed to IBM’s approach. WA has a very rigid dialog management environment. Elements like disambiguation, digression, multiple conditioned responses and a huge amount of business logic are vested in the dialog, and makes for a complex environment.

Slots are also defined in the dialog. Most probably the best approach will be to create user stories from the different permutations which exist within the defined WA dialog. This will be a manual process; with the added advantage of being able to visually inspect the dialog within WA.

Conclusion

To constitute the Rasa NLU portion will be very much straight forward; the NLU API if you like. Most, if not all, of this portion can be automated with a text manipulation program. The Rasa Core portion will take manual intervention.

However, by stepping through the application to extracting the user stories can also be used as an opportunity to optimize and improve the chatbot dialog.


Read More Here…

Cobus GreylingHi, I’m Cobus… Currently I conceptualize, mock-up, wire-frame, prototype and develop final products. Primarily…cobusgreyling.me

Cobus Greyling – MediumRead writing from Cobus Greyling on Medium. NLP/NLU, Chatbots, Voice, Conversational UI/UX, CX Designer, Developer…medium.com

Rasa as open source alternative to IBM Watson – Migration TipsThere is no support for IBM Watson yet. However, a group of community members is working on a way to use exported IBM…rasa.com

Rasa: Open source conversational AIBuild contextual AI assistants and chatbots in text and voice with our open source machine learning framework. Scale it…www.rasa.com