Considerations When Measuring Chatbot Success

Considerations When Measuring Chatbot Success

And What Principles You Should Implement…


Performance measures are important to organizations wanting to track their investment in a conversational interface…

But standards & metrics differ by industry and obviously by companies in each industry. Due to the nascent nature of the technology companies are also eager to learn from one another.

With some overestimate the importance and impact of their chatbot, and other heaving discounting the significant impact their conversational interface is having…

Industry Type Matters

Call Deflection

Obviously chatbots are implemented across a vast array of industries. These industries use different parameters. Parameters which they deem as crucial to the measuring of success in their environment.

Microsoft Power Virtual Agents have Analytics Built In

Banking and financial sectors use chatbots to perform existing tasks faster. And an important driver is lower call volumes and how much savings are incurred from call deflection.

Quality Conversations

The most common and probably important chatbot performance metric is conversation length and structure. In most cases conversation transcripts are reviewed and manually classed in order so points of improvement noted.

Organisations are aiming for shorter conversations and simplified dialog structures. A conversation or specific dialog always have a happy path which developers hope the user will find and stick to.

Digression in a Chatbot Conversation

A rudimentary and simplistic approach would be to have a repair path, or a few. Paths which intends to bring the conversation back to the happy path from points of digression. Hence ‘repairing’ it.

This approach might lead to a situation called fallback proliferation.

Three elements which will improve the quality of the conversation are:

Channel Retention

For others, channel retention is an important metric in measuring theirs chatbot’s success. Hence customers or users returning to be serviced via the chatbot channel.

If a company aims to replace traditional communication channels (especially telephone calls) the goal is to have retention in the chatbot medium. The argument is that if the experience was good, to exceptional, the user is bound to return.

Added to this, the call center savings are very tangible.

The caution here is that there are other self-help channels too, other than calling an agent. User interactions must be viewed in its totality in terms of mediums available to the user.

An additional measure of chatbot performance is the ability and degree to which the experience is personalized. Only relevant information should be surfaced to the user.

Batch Test Datasets — Use Batch Testing to find utterance prediction issues in your app and fix them.

It may again vary based on industry. Banks look at relevant options to the specific user.

Retail might focus on ways to get users to shop. With the chatbot presenting option as a shot assistant would.

Brands transform themselves into personalized shopping assistant.

Pillars of Trust

In general, a user experience trust manifests as a firm belief in reliability, truth or the ability of someone or something to act for the their good. Among us as humans, trust is seated in credibility and confidence in another’s judgement.

Also part of this is predictability of behavior.


  • The belief that the bot can deliver on expectations and the extend to which the bot is seen as effective.
  • The ability to not only survive but compete in a marketplace of mediums and channels.


  • This is defined as user held belief that that organization is fair and just.


  • That the organization acts consistently and dependably.


The emphasis on trust is heightened in certain environments.

Environments such as financial care, heath care and other fields demanding sensitive data by which the user may be exposed to physical, financial or psychological harm. User will not share personal information if they are unsure about security.


A crucial part of trust is related to anthropomorphizezation. Ascribing human attributes to a chatbot. This manifest in the chatbot having a name, a “voice”, personality etc.

A tool to create & edit training examples for Rasa NLU

We as humans have been placing social expectations on computers for a while. And, athropomorphization increases users’ trust in computer agents.

High quality automation may lead to more fruitful interaction because the machine seems to be more objective and rational that a human.

Humans tend to be less sympathetic toward chatbots; because humans are expected to be imperfect while the opposite is true for automation.

An important point to note is that anthropomorphization is not about creating superficial human-like characteristics. But rather a chatbot which is human like-minded.


I would like end with this quote from Zadrozny et al.

Penned twenty years ago…

Customers have the initiative to express their interests, wishes, or queries directly and naturally, by speaking, typing, and pointing.

The computer system provides intelligent answers or asks relevant questions because it has a model of the domain and a user model.

The business goal of such computerized systems is to create the marketplace of one.

In essence, improved discourse models can enable better one-to-one context for each individual.

Even though we build NL systems, we realize this goal cannot be fully achieved due to limitations of science, technology, business knowledge, and programming environments.

Read More Here…

Cobus Greyling – MediumRead writing from Cobus Greyling on Medium. NLP/NLU, Chatbots, Voice, Conversational UI/UX, CX Designer, Developer…

Cobus GreylingHi, I’m Cobus… Currently I conceptualize, mock-up, wire-frame, prototype and develop final products. Primarily…

In bot we trust: A new methodology ofchatbot performance measuresChatbots are used frequently in business to facilitate various processes,particularly those related to customer service…

Cobus Greyling – Chapter Lead: TOBi Chatbot Design – Vodacom | LinkedInNLP/NLU, Chatbots, Voice, Conversational UI/UX, CX Designer, Developer, Ubiquitous User Interfaces. CX requires four…

Leave a Reply

Your email address will not be published. Required fields are marked *