Microsoft's AI shopping announcement contains hallucinations in the demo

Product search online has gotten hard: Google is full of spam and Amazon is overrun with fakes. At PerfectRec, a product recommendation engine, we think that’s a real problem. Apparently, so does Microsoft. 

A few weeks ago, Microsoft announced their latest foray into e-commerce search: AI-powered buying guides in Bing. We were curious to dig in and see just how well (or not) this feature performed, since the problems with large language models like ChatGPT is that they tend to make up fake information – errors called “hallucinations.”

It turns out we didn’t have to look very far. In fact, Microsoft’s own promotional materials include hallucinations about headphone quality. (Wayback Machine link, in case MS updates their blog post.)

Here’s one of the animations Microsoft included in their launch announcement. 

And here’s a screenshot of the hallucination:

So what’s the issue here? 

Responding to a prompt, Bing says the Surface Headphones 2 are “the best ANC headphones.” That’s not true, at least not according to any human reviewers we can find. 

• The Sound Guy’s review says: "The Microsoft Surface Headphones 2 is cheaper than many of the best consumer ANC headphones that Sony and Bose have to offer.” This contrasts with Bing’s contention that they’re “expensive but worth it.”

• The Verge’s review is more explicit, stating: “Microsoft’s noise canceling isn’t quite as effective as what Bose or Sony can achieve.”

• RTings says the noise canceling on the Surface Headphones 2 doesn’t even live up to its predecessor: “The first-gen can block out slightly more noise.”

• The Guardian says “noise canceling and sound not quite as good as rivals.”

• Laptop Magazine didn’t include them in their list of the best 13 noise canceling headphones. 

• The Wirecutter doesn’t include the Surface Headphones 2 in their list of ‘Best Headphones’ either. 

In the previous chat message in the demo, Bing AI lists the New York Times as the top source for its headphones buying guide, so we checked to see if maybe that’s where the claim came from. But it turns out the paper of record has never mentioned the Surface Headphones 2 in any context. 

So where did Bing’s AI shopping get the idea that the Surface Headphones 2 were “the best ANC headphones”? We’re not sure – but the Surface headphones are, of course, a Microsoft product. Bing’s affection for them isn’t a great sign that its AI Buying Guide will give accurate or objective results. 

We spotted this problem with Bing because PerfectRec has been looking at ways to use AI to explain our own product recommendations. But the problem we keep encountering is that today’s LLMs are constantly hallucinating about basic facts and making false claims about how products compare with one another. It looks like Microsoft hasn’t solved this problem either. We’ve held off on releasing our LLM-based feature until the hallucination issue is better under control. Microsoft apparently came to a different conclusion.