The conventional retail chatbot, in the form that has dominated the market since the first wave of large-language-model integrations reached commercial deployment in 2023, has acquired a reputation for unreliability that is, in my observation, broadly deserved. The chatbot is presented to the visitor as a knowledgeable representative of the retailer; the chatbot, in its actual implementation, is a general-purpose language model with little or no grounding in the retailer’s own documentation; the chatbot, as a consequence, generates answers that are confidently expressed and frequently incorrect. The reputational cost to the retailer of an answer that is publicly wrong is non-trivial; the operational cost of the customer-service follow-ups that the wrong answers generate is also non-trivial; the conversion uplift that the chatbot produces is, in the data I have available, often negative.
AI Interactive FAQ is a tool I built to address the failure mode by abandoning the general-purpose chatbot pattern entirely. The tool does not present itself as a knowledgeable representative; it presents itself as an interface to the retailer’s existing documentation. The questions it answers are answered from the documentation alone; the questions it cannot answer from the documentation are routed to a human responder rather than guessed at. This post is an account of why the conventional pattern fails, what the alternative architecture looks like, and what the conversion data has, in the eighteen months since the tool’s first deployment, told me about the cases in which the alternative is worth the implementation cost.
Why the general-purpose chatbot fails on retail catalogues
The failure mode of the conventional chatbot is well-documented in the language-model literature and is sometimes referred to as hallucination, although the term is somewhat misleading. The model is not, in any meaningful sense, hallucinating; it is producing the most statistically plausible continuation of the input text given its training distribution. For questions about general knowledge — the height of a mountain, the population of a city, the year a film was released — the most statistically plausible continuation is also, in the great majority of cases, the correct answer, because the training data includes a great deal of accurate information on these topics. For questions about a specific retailer’s specific products, policies and procedures — the warranty period for a particular model, the return window for a particular category, the dimensions of a particular variant — the training data includes essentially nothing accurate, and the most statistically plausible continuation is the answer that the model would expect to be true given the general patterns of retail commerce. The expected answer and the correct answer are, on retail catalogues with any meaningful specificity, frequently different.
The mitigation that the chatbot vendors most commonly offer is to fine-tune the underlying model on the retailer’s documentation. The mitigation is real but is, in my experience, considerably less complete than the vendors’ marketing material suggests. A fine-tuned model is biased towards the retailer’s documentation but is not constrained by it; the model continues to answer questions for which the documentation provides no information, and the answers continue to be statistically plausible rather than correct. The fine-tuning reduces the rate of incorrect answers; it does not, in my observation, reduce the rate to a level at which the chatbot is operationally safe to expose to customers without supervision.
The retrieval-augmented architecture
The architecture that AI Interactive FAQ implements instead is a retrieval-augmented generation pattern, sometimes referred to in the language-model literature as RAG. The architecture has three components: a document store containing the retailer’s documentation, decomposed into small passages and indexed for similarity search; a retrieval layer that, given a user’s question, identifies the passages from the document store most likely to contain the answer; and a generation layer that, given the user’s question and the retrieved passages, produces a natural-language answer constrained to the content of the passages.
The constraint is the operative property of the architecture. The generation layer is instructed, via a system prompt with several layers of explicit guardrails, to answer the user’s question from the retrieved passages alone; if the passages do not contain the answer, the generation layer is instructed to say so and to offer the user a route to a human responder. The instruction is, in the great majority of cases, followed; the small fraction of cases in which it is not followed are caught by a post-generation verification step that checks the answer against the retrieved passages and rejects answers that do not have textual support.
The architecture is not architecturally novel; the retrieval-augmented pattern has been documented in the language-model literature since at least 2020 and is now sufficiently mature that several open-source frameworks implement it directly. The aspect that is novel, in the context of retail commerce, is the strict enforcement of the no-answer fallback. The conventional chatbot is rewarded by the vendor’s metrics for producing an answer to every question; the retrieval-augmented architecture is rewarded for producing an answer to the questions it can answer correctly and for routing the rest to a human, which is the behavioural target that operational safety requires and which the conventional pattern explicitly works against.
The document-preparation work that nobody discusses
The principal cost of the retrieval-augmented architecture is not the language-model integration itself, which is by 2026 a routine piece of engineering, but the preparation of the document store. The retailer’s existing documentation is, in essentially every case I have worked with, in a form that is not directly suitable for retrieval. Product specifications are scattered across CMS pages, PDF brochures, third-party catalogue feeds and informal notes in the customer-service team’s ticketing system; warranty terms are sometimes documented and sometimes not; return policies are sometimes documented in one form on the public site and in a different form in the internal training material; the dimensions of products are sometimes correct and sometimes inherited from earlier model years that the documentation has not been updated to reflect.
The retrieval layer can only return what the document store contains; if the document store contains incomplete, inconsistent or out-of-date information, the generation layer’s answers will reflect that. The implementation work for AI Interactive FAQ on each new deployment has therefore included, in addition to the technical integration, a documentation audit and consolidation exercise that has typically taken between two and four weeks of the retailer’s content team’s time. The exercise is rarely budgeted for at the outset of the project; it is, in my experience, the single most reliable predictor of whether the tool will produce a useful conversion outcome.
The conversion data
The data I have collected from the deployments of the tool is, in summary, that retrieval-augmented FAQ produces a conversion uplift of approximately four to six per cent on the visitor population that interacts with the widget, measured against a control population that did not see the widget. The uplift is smaller than the headline figures sometimes claimed for guided-discovery tools or product configurators; it is, however, considerably more reliable, in the sense that essentially every deployment of the tool produced a measurable uplift, whereas the larger figures from the other tools are concentrated in a smaller number of well-suited deployments and absent from the rest.
The uplift comes, on the data I have available, from two distinct sources. The first is the resolution of pre-purchase questions that would otherwise have caused the visitor to abandon the session — questions about delivery times, warranty terms, sizing or compatibility. The second, somewhat unexpectedly, is the surfacing of relevant products in the answers to questions that were not, on their face, about which product to buy. A visitor who asked the widget about the warranty terms for a particular product was, in the implementation, presented with the answer alongside a link to the product itself; a non-trivial fraction of the visitors so presented proceeded to view the product who would not, on the basis of the navigation paths their session had previously taken, have done so otherwise. The second source of uplift is, I suspect, an under-appreciated property of the architecture and one that is worth optimising for in subsequent versions.
What the tool does not do
The tool does not, in its current form, handle multi-turn conversations beyond the immediate question-and-answer pair. The visitor asks a question; the tool answers; the visitor can ask another question, but the tool does not maintain a conversational context across questions and the second question is answered as if the first had not been asked. The decision to constrain the tool to single-turn interactions was deliberate; the conversational chatbot pattern is the pattern that produces the hallucination failures most reliably, and the constraint is the most direct way to avoid the failures. There exist deployments in which a multi-turn pattern would be valuable; for those deployments, a different architecture is appropriate and the tool is, by design, not the right choice.
An advisory close
The case for AI in retail customer interaction is, in my view, considerably stronger than the dominant procurement narrative suggests; the case is also considerably narrower. The general-purpose chatbot is the product of a procurement narrative that is more concerned with the appearance of innovation than with the operational outcomes of the deployment, and the operational outcomes have, in the data I have available, been disappointing. The retrieval-augmented FAQ is a more modest product with a smaller surface area; it is, on the basis of the deployments I have shipped, the form of AI integration that retailers are most likely to recover their investment from. (For a related view on the broader procurement narrative around AI search, see AI search is overhyped: auto-complete is underbuilt.)
It is recommended that retailers considering AI integration begin with the question that the tool is being asked to answer rather than with the technology that the integration uses. The questions that the retailer’s existing documentation can answer correctly are the questions that retrieval-augmented architecture can answer correctly; the questions that the documentation cannot answer correctly are the questions that no architecture can answer correctly without first solving the documentation problem, which is the problem that the procurement conversations rarely begin with and which determines, in my experience, whether the project ships as a success or a failure.