An assistant that can't say "I don't know" is a problem

In short

• In a store, a made-up answer isn't a charming mistake: it's a wrong delivery date, a false promise of compatibility, a return.
• A useful assistant answers from the store's own sources, and not from the model's memory.
• When there's no source, the right answer is to say so and hand over to someone who knows. That's the feature, not the failure.
• What separates an assistant from a generic chat isn't the model: it's the sources, the limits and what happens when it doesn't know.

There's a question that almost never comes up in demos of shopping assistants: what does it do when it doesn't know? That's a shame, because it's the question that decides whether an assistant can stay in a serious store.

In a general-purpose chat, a wrong answer costs a smile and a correction. In a store, it costs something else. If the assistant says that charger is compatible with that laptop and it isn't, someone pays return postage, writes an annoyed email and doesn't come back. The conversation ended well, the sale ended badly.

What confines the answers

The difference between an assistant that serves a store and one that embarrasses it is almost entirely in the sources. In our case there are three, in this order:

1. The store's knowledge base. The answers written by the team: opening hours, lead times, exchange policy, delivery quirks. When a written answer exists, that's the one that counts, even if it contradicts everything else. The reason is practical: the terms of sale were written once, for a lawyer; the team's note was written this week.
2. The store's indexed pages and articles. Terms and conditions, corporate pages, blog articles, product pages. It's what the store has already published and stood behind.
3. The catalogue, through the search. Recommended products are always products the search returned, with the price and the stock that are in the catalogue at that moment.

What is left out matters as much as what is kept in: the assistant doesn't answer from memory about how the store works. If the question is "do you deliver to Madeira in 24 hours?" and nothing is written about it, the answer isn't a plausible estimate. It's to say it doesn't have that information and to offer the contact of someone who does.

Generic assistant and store assistant

The fourth row is the one that usually decides the purchase of the tool, and it's the easiest to test in a demo: ask something the store hasn't written down anywhere and see what happens.

Saying "I don't know" without losing the sale

There's a legitimate objection to all of this: an assistant that says "I don't know" often is useless and irritating. The objection is fair, and the answer isn't to answer anyway. It's to do three things before giving up on the conversation.

• Ask back. Many questions with no answer are badly framed questions. "Will it fit me?" has no answer; "what height and what weight?" does. It's the principle behind AI-assisted purchase, where each answer narrows the catalogue instead of guessing.
• Offer the closest thing. If there's no answer about compatibility, there is the list of products the store has for that model, with the warning that compatibility isn't confirmed.
• Hand over with context. Passing to a person is the right way out, and it's far better when the person gets the whole conversation instead of "a customer wants to speak to you". It's worth deciding in advance where to: a channel, opening hours and the usual response time said out loud are worth more than a "contact us", and that's what Buskara asks for in the customer support settings.

What we never do is the fourth option, which is the most tempting: give the likely answer and leave the customer to confirm it on their own.

Pros and cons of putting an assistant in the store

The cost line deserves a note, and it's a difference of kind and not of degree: a conversation costs more than a search, always, and that's why the Buskara plans count the two things separately. Which is why the assistant shouldn't be the way in for everything. The way in is the search, which is cheap and solves most of it; the assistant comes in where the search doesn't reach.

Where it doesn't reach is measurable, and there's no need to guess: it's the searches that end with no results written in the form of a sentence, and the service questions that turn up in the search box. In Baymard's self-service testing, 34% of participants tried to search for content that wasn't a product, and that's the search category sites serve worst of all.

What the store has to write

The part nobody likes to hear: the quality of the answers depends more on what the store has written than on the model chosen. An assistant connected to a store with no exchanges page will answer badly about exchanges, with any model.

The minimum worth writing, in order of payback:

1. Delivery times and costs, including the exceptions the team repeats every day.
2. Exchange and returns policy, with what happens to someone who bought in a sale.
3. Size or compatibility tables, if the catalogue has them.
4. The five questions customer support answers most often by email.

It's a few hours of work, done once, and it's what separates an assistant that closes sales from one that produces pleasant conversations with no consequence.

There's a shortcut for knowing what to write, and it's reading what has already been asked. Half an hour in the conversation logs yields more good entries than an afternoon imagining them at your desk, because the questions are written in the words of the people who asked them.

Frequently asked questions

Can an AI assistant make up answers about my store?

It can, if it answers from memory. An assistant confined to the store's sources (catalogue, published pages and knowledge base) only answers from what it finds, and when it finds nothing it should say so instead of estimating.

Does the assistant recommend products that are out of stock?

It shouldn't. Recommendations have to come from a search of the catalogue at the time of the conversation, with the stock and the price of that moment. A product list prepared in advance ages and starts recommending what no longer exists.

What's the difference between the assistant and the search box?

The search answers people who know how to name what they want, and it's cheap and instant. The assistant answers people who describe a situation or ask a question, it costs more per use and it should come in where the search doesn't reach, not before it.

What happens when the assistant can't answer?

It should ask back to frame the question better, offer the closest thing there is with the appropriate warning, and hand over to a person with the whole conversation attached. What it shouldn't do is answer with what's likely.

What do I need to write for the assistant to work well?

Delivery times and costs with the exceptions, exchange and returns policy, size or compatibility tables, and the questions customer support answers most often. It's a few hours of work and it's what changes the quality of the answers most.