Testing chatbots: An introduction to conversational interfaces

I’m lucky to have gotten involved in a project testing chatbots. It’s an emerging technology, so I’ll do a series of posts on different aspects of testing chatbots.

You probably already know plenty of chatbots

Chatbots, conversational interfaces, digital assistants.. they have many names. A chatbot is, in short, a program that can communicate with a user. The user writes or says something to the chatbot, which then analyzes and finds/calculates an answer to the sentence.

The best known chatbots are probably Amazon’s Alexa, Apple’s Siri, or Google’s.. Google. They are bots that are part of your phone or speaker. They can perform tasks like searching for answers to a question, playing music or ordering something. But you have to ask them in the right way.

The last time I flew with KLM airlines, a chatbot updated me on my flight, and I could ask it to do basic tasks like changing my seat. It worked really well, though it couldn’t help me with more complex tasks, like when my flight got cancelled.

testing chatbots
I only used the KLM chatbot for flight information, but it can also help with travel-related stuff like what to pack for the trip. Clever.

Why use chatbots?

So the chatbots are already here, but in the next couple of years, we’re going to see a lot more of them. Frameworks for them are both affordable and easily adaptable to specific companies. The chatbot can become part of a company’s support or information departments, without having to be built from scratch.

But why would you want to use a chatbot in the first place?

Because chatbots can substitute people! Bam, right in the middle of all the automation-fear that’s going on at the moment.

But even if it sounds a little harsh, the idea is actually quite nice. So let the me rephrase that: Chatbots can unburden people by taking care of the boring, basic routine cases that don’t require an emphatic and brilliant human mind.

Most companies already offer live chat with a human being on their website as an instant contact method for quick questions. A chatbot could easily take care of most of the questions I’ve asked on a website’s live chat channel. Instead of using time and energy on basic tasks, people working in a support department can focus on challenging cases where they make a difference. That makes for happier, more effective workers.

testing chatbots

Testing simple chatbots

When it comes to testing chatbots of the more basic kind, you basically need to test how it answers different questions.

  1. Does the bot know the answers to questions it will encounter.
  2. What does the bot answer to questions it might encounter?
  3. What does the bot answer to questions it probably never will encounter?

Users will probably say “Hi” to the chatbot, so there’s good reason to have that as part of its script, along with all the other highly possible scenarios. In the other end, the chatbot will probably not encounter anyone who starts talking about that their dog just died. But if they do, the chatbot should at least let them know that the question is outside its area of expertise.

testing chatbots
Poncho the weather-chatbot isn’t good for talking about feelings

Sounds simple, right?

It’s not. It’s really content-heavy. Consider all the different ways people might ask a question. Then add spelling errors and auto-correct on top of that. If the questions alternate just the slightest from the bot’s script, users will get the standard “I don’t understand the question” reply. That either makes them close the chatbot, yell, or leaves them desperately trying to figure out the right way to ask. So you need to get some data on what your users are going to ask the chatbot. If you don’t, you’ll be doing a so-called cold-start. Working from that data you slowly and thoroughly add new forms of the same question to the bots database. It’s a tedious job.

Luckily the emerging “deep learning” or “machine learning” technology offers different solutions to that. I’ll take a look at that, and discuss how on earth you can test it, in a following post.

Wanna try out testing chatbots?

It’s always fun to play around and see what their limits are. It gives you a good understanding of what chatbots are and how they function.

  • Mitsuku: Voted most human chatbot in 2013 and 2017. Don’t let the outdated website fool you.
  • Rose – Brilling Understanding Inc: My first attempts at communicating ended up in me being trashtalked. Apart from that, Rose knows a lot about security.
  • Poncho is a weather bot in Messenger. It can tell you about.. well.. the weather.


Kontakt Gheist

Kaffe? (Eller the)