"It's only answering 32% of customer queries: Why is Conversational AI so bad?"

That's the question I got this week from a despondent Chief Retail Officer talking about their bank's deployment of a Conversational AI solution.

"Oh, that doesn't sound too promising?" I said, encouraging the executive to say some more.

A pained look swept across the executive's face.

"Yeah, they [the vendor] promised it would be in the high 80s from day one, it's just a total disaster. Customers hate it. Our employees hate it. It's actually creating more work for us internally, cleaning up its mess."

I empathised. This is not good news for the Chief Retail Officer - there are bonuses and expectations on the line throughout the retail division and the chatbot was meant to really impress.

"Who did you use?" I asked. They hadn't given the name of the vendor.

"Oh, we used [redacted]."

Now, I can't actually remember the name of the vendor, but – well, that says it all – it wasn't a vendor you or I have heard of.

I'm willing to bet they are not on your radar.

The executive brought up the company's site on her iPhone and showed me briefly.

"You haven't come across them?" they prompted.

"Er, no sorry," I had to reply.

They looked really downbeat.

"Who should we be using? Does this stuff actually work?" They asked.

I assured them that with the right vendor, Conversational AI can be incredibly powerful.

As for a vendor they should use, I rattled off a few brand names and the executive jotted them down.

I'm astonished this bank got all the way through to deployment and production live with such a vendor. What's fascinating, though, is that the executive was keen to stress that the vendor was offering "Generative AI" as if that was a safety blanket.

I wouldn't be surprised if the vendor just downloaded a copy of Meta's Llama, 'trained' it on some bank-supplied PDFs and threw it up on the website for customers to start using. No wonder it's not that effective.

So, if you're sitting wondering if your marketing and sales teams are doing a stellar job: No. Not in financial services, anyway.

This executive couldn't name one single Conversational AI vendor, beyond the one they were using.

I find this frustrating and alarming. What it shows is that – in this market sample size of one – not a single Conversational AI vendor has managed to reach this Chief Retail Officer with any messages material enough that they can remember or recall a single brand.

That in itself isn't necessarily a problem, but it's not a good situation for the wider marketplace though.

It means it's all sitting there, waiting for someone to get it right.