Chatbot Testing: How to Get it Right in the First Go

Chatbot Testing Best Practices

Chatbots have forayed into the business arena with a bang. Enterprises have been leveraging chatbots to not only engage the end-users, but also study their behaviour and habits. It is now established that a chatbot does much more than just messaging.

Residing within messaging platforms, chatbots are rapidly adding value to conversations. Chatbots now have a context driven by intelligence. The conversations aim at solving customer problems in the most convenient manner. The entire experience, thus, depends deeply on how tactfully the chatbot steers conversations to yield the most acceptable responses.

Conversation is the key skillset of a successful chatbot. An engaging bot is one that can understand the customer’s questions and respond promptly. But this is just the tip of the iceberg. Businesses need to thoroughly focus of enriching the user experience delivered by bots – as they emerge as one of the most promising digital touchpoints in a customer’s journey.

So, we need to establish what ground-rules should be established for this new paradigm. Earlier, we talked about the qualities of the right chatbot development partner and gave a step-by-step approach to building a remarkable chatbot personality.

In this current article, we will talk about the must-haves in your chatbot testing checklist. We will also guide you on how to go about these checks and balances.

1. Testing a Chatbot’s User Experience (UX)

The user experience of a chatbot is comprised of various aspects. It includes the visual appearance and the personality of the chatbot during conversations. While testing out your bot’s performance from the UX point of view, you must consider the following:

The Onboarding Process

Verify that onboarding is the first thing that the bot does when the user opens the chatbot window on your channel. For example, on Facebook, a greeting screen and greeting text should appear the moment the user clicks on the chatbot option. This allows the user to understand what the next step is. A ‘Get Started’ button can be a good idea for those who are conversing for the first time.

You need to establish the ground-rules before the conversations starts. This feature gives the user a fair idea about what the bot will do.

Developer Tip: Consider this feature during the time of bot’s development. Fill up on some onboarding tricks.

Tester Tip: Ensure onboarding in the chatbot. This will help you decide whether the user will stick around or go away. Take a look at the example below:

slackbot - onboarding example

Using Generic Templates

The use of generic templates is a great way to ensure that the user feels connected while finding your bot on different channels. Generic templates help with better engagement. Your content cards must appear more or less the same across multiple channels to retain the user’s attention.

Developer Tip: Check JSON templates provided by channels and compare.  Use those templates that allow you to make a content card with thumbnail, text, or title with minimum of channel-specific properties.

Tester Tip: Make sure you check multiple channels to validate the content cards.

Developer Tip

Balancing Out Text Length

You need to be mindful of the fact that it’s a conversation. Bot responses should be concise, specific and not a bunch of long strings text. Text responses should not be broken. For example, channels keep a word-limit as to how much can be included in a single response. If it exceeds this limit, they break the text into two parts, which looks hideous.

Developer Tip: Take into account a channel’s text limits for each message. Use paging when necessary.

Tester Tip:  Verify that there is no wall of text or any broken text.

Good text length in chatbot example

Ensuring Correct Link Navigation

The connector links included in a bot’s conversation must be checked thoroughly. The links should not be broken or redirect users to incorrect pages. This may otherwise leave the user with a terrible experience. He/She may end up never using your bot again.

Tester Tip: Ensure that the ‘back button’ immediately takes the user back to bot interface.

Including Feedback Options

A user would always love to share his experience, good or bad, with you. This will help you in precisely knowing the experience delivered to a specific user. Make sure a feedback point or “contact us” option is included with the bot. This small touchpoint within the bot can often be a saviour for you.

Developer Tip: Feedback storage in the database should be part of your bot development.

Tester Tip: Verify if this feature is easily visible to the end-user. You must check it thoroughly with single-line and multiple-line inputs.

Including Feedback Options

User can put in “Very Likely” or “Likely”. “AskHaley” also recommends to share the bot. If a user isn’t happy with the product, Ask Haley solicits open-ended feedback to learn more.

feedback option in chatbot

Testing Conversational Flow

This section deals with the context of the conversation. An appropriate conversation flow is one that talks tactfully and keeps the user engaged with relevant replies.

Ensuring a Problem-Solving Approach

Here, you need to examine if the bot is serving its core purpose. For example, if it’s a news bot, you need to test whether it is delivering the news properly. Or, if it is a conversational-commerce bot, whether it is able to search and proceed to checkout smoothly.


Checking Context Awareness

The bot needs to be intelligent enough to remember specific talking points. Consider this example:

BOT: Shall I arrange a callback

User: Ok / Yes / Sure

Here, you need to ensure that your bot is able to efficiently understand that the user wants a callback.

Developer Tip: Make sure you maintain a context-stack in your conversation to keep track of important points during the conversation.

Tester Tip: Test by throwing off random words in the middle of a flow or leave the chat and then return to see if the bot is able to maintain the same flow.

Adding Alternative, Yet Valid, Responses to the Same Text

To break the monotony in a chatbot conversation, it is important that you pep it up with interesting responses to basic messages. Test whether expressions like, ”Hi, Hello….” yield different responses from the user. The bot could sometimes say, “Hello,” sometimes “Hi,” or “Hey.” This will add life to the conversation.

Developer Tip: Configure multiple responses to common greetings or sign-off words.

Tester Tip: Test your bot by throwing off these common words in the middle of a conversation. See if the conversation breaks, or if the bot smartly takes the conversation forward.


Including Pleasant Responses to Erroneous User Inputs

Users often tend to throw off the bot with challenging questions, assuming that it will magically understand everything. Here, your bot is at a real test. If it is unable to understand the user input, it must clearly state it. But saying this pleasantly is crucial to win the appreciation of the user. Make sure the bot handles such erroneous inputs in the friendliest manner. It may say:

 “I am sorry I am unable to understand you!”

“Our Oompa Loompas are unable to understand….But here’s what they can do…..<show menu>”



Including Help Features

Every user is different and looks at the bot from a different perspective or expectation. Thus, a help center is a crucial feature within the bot. This will ensure the user does not drop out in the middle of the conversation. It is good idea to introduce the help feature during the onboarding process.

Including Help Features


3. Testing a Chatbot’s Intelligence Quotient

A chatbot has to exhibit a personality. It has to be witty, quick, and humorous to be called intelligent. So, this is one area where you need to make unfathomable efforts. Your bot must impress the user with its responses.  This will involve the following aspects:

Sharpening Your Bot’s Memory

You need to test whether your bot remembers everything discussed earlier in the conversation and does not expect the user to type in the same inputs again.

For instance, if the conversation involves the bot asking the name of the user, you need to check if it remembers the name. You also need to check if it remembers where the user left off in the conversation.

This feature will also eventually help you expand the scope for your bot and deliver a more personalized experience to the user.

Developer Tip: Make sure to test state-variables in the bot with multiple users. Check for variable-state safety with multiple conversation.

Tester Tip: List the data maintained in bot and test it with multiple conversations to see if your bot remembers everything you shared with it.

Enhancing the Bot’s Ability to Validate the User’s Intent

Validating the user’s intent implies that the bot should be able to understand exactly what the user wants to know. This relates to the moment where the bot is able to deduce what the user wants.

There are two kinds of bots – Rule based and Artificial Intelligence (AI) based. There are some bots which do not have a ”typing option” for users; they simply guide the user through the conversation while providing options for users to click. Bots that involve a typing option need to validate the text. They need to check if the user has asked the same query multiple times, and whether there are some typos or ambiguity.

Developer Tip: Either use buttons to present options, or maintain a list of intents in conversation code to maintain the fluidity of a human conversation.

Tester Tip: List down the main intents of the conversation flow and present them to the bot in multiple ways. This will help verify if the bot is able to recognize the intents in conversation.

Bot validating user intent

4. Chatbot Testing Response Time

Bots only involve conversational texts. So, the response time is a crucial factor that drives the user engagement. The responses have to be prompt.

Market surveys have shown that waiting during customer care calls has been one of the biggest frustrations customers have. The reason for this, mainly, is the waiting time. The customer will lose interest the moment he or she is asked to wait for the next response.

Hence, a bot, seen as a replacement for customer care representatives, simply cannot afford to keep the user waiting for its response.

5. Testing a Chatbot’s Security

A chatbot has access to sensitive data, as the responses need to be linked to customer information. This elevates security threats, too.

Masking Sensitive Data

Chatbots reside inside our social messaging platforms and has access to our sensitive data. This data includes the user’s personal information. A banking bot, for instance, may ask for sensitive info from the user. Imagine if the bot ends up exposing some of that critical piece of info during a future conversation. You cannot let this happen. Here’s what you need to ensure that info is secured:

  • Create a mechanism to ensure controlled exposure of sensitive info
  • Secure where sensitive data is stored

Testing for Outages

Testing for Outages

There are two aspects to this factor. One is killing the bot with vulnerabilities in the technologies used. The second is the old-school method: “spam the webhook with spoofed messages.”

Nowadays, most of the technologies are pretty mature and most of the security glitches are fixed quickly. But due to the initial boom in bots, there was a mushrooming of bot-libraries.

As I have worked with Facebook chat bots mostly, keep in mind the implementation of ”X-Hub-Signature” in any library you use. Make sure it is implemented the right way and not skipped. As this can allow anyone to spoof the request and make your bot believe that it’s from Facebook.

Check this on GitHub: It is a metasploit module for testing spoof of messages or a ‘way to kill your bot.’

Take a look at the following example from Facebook:


6. Testing a Chatbot’s Multiple Channel Compatibility

Multiple channel compatibility is another key component when testing the bot. There are multiple messaging channels and each has a unique way of technical implementation. So, if your bot is to live inside multiple channels, make sure it’s a happy living.

This factor allows you to validate and review the end-user’s experience across multiple channels like Facebook, Telegram, Kik, etc.

Now, channels allow developers and testers to integrate the bot into a channel for testing purposes.

Check whether your images and templates (content cards) appear correct. Ensure the following:

  • The button text appears properly
  • Images appear in the right place and don’t get cropped across different channels
  • The ”bot responses” appear the same across multiple channels


In a nutshell, you should validate the above checkpoints as part of your regular bot-testing plan and I am sure you will create something that will make the end-user happy. Regular testing of chatbots is the key to their success.

Gurpreet Singh

About the Author

Gurpreet Singh has a rich experience in Microsoft Technologies. He loves to work on RnD tasks and emerging and innovation-oriented technologies. Chatbot is one of his key areas of interest. During his personal time, he likes to read technology blogs.

Leave a Comment

Pin It on Pinterest


Articles written by industry experts about things that matter most in designing and building Digital Products and Platforms for Startups and Enterprises.

Subscribe to our

Digital Insights

Follow us on:

Aw, yeah! That was a smart move.

We have sent a short welcome email your way.