Every AI chatbot vendor (ourselves included) tells the same story: “connect your knowledge base and the assistant will answer with your information”. The story is true. What almost never gets said is the fine print: the assistant will answer with your information exactly as it is, and in most companies, exactly as it is means outdated, contradictory and incomplete.
A language model doesn’t fix bad documentation: it amplifies it. Where there used to be an outdated PDF almost nobody read, there is now an assistant reciting that PDF with total confidence to every customer who asks. An assistant’s quality has a ceiling, and that ceiling is your documentation.
After connecting assistants to knowledge bases across very different sectors, these are the ten problems we find over and over. Use them as an audit: if you tick more than three, you have work to do before (or alongside) the chatbot.
The checklist
1. Are there contradictions between documents?
The classic: the website says 30-day returns, the terms-and-conditions PDF says 15, and the internal FAQ says “check with the manager”. A human resolves the contradiction with context; a semantic search retrieves any of the three versions depending on how the question is phrased. Quick test: pick your five most important policies and search for them across all your content repositories. If different versions show up, the assistant will serve all of them.
2. Is the content dated and versioned?
Without dates, nobody (not you, not the assistant, not whoever audits an error) can know which version of a policy was in force when. Every document needs an owner, a last-reviewed date and a next-review date. It’s boring, and it’s the difference between being able to audit an answer and not being able to.
3. How much knowledge lives only in people’s heads?
Try this exercise: take your support team’s last 50 conversations and check how many of the answers your team gave are written down in any document. The usual result is surprising: a large share of the real knowledge (the exceptions, the edge cases, the “actually, we do it like this”) isn’t documented anywhere. That tribal knowledge is invisible to any AI. Documenting it is the most valuable work on this entire list, and nobody can do it for you.
4. Do your real frequently asked questions have written answers?
Related but different: your public FAQs tend to answer the questions marketing would like to receive, not the ones support actually receives. The source of truth for what your customers ask is your ticket inbox, not your FAQ page. Cross-reference the two and write what’s missing.
5. Is the content in processable formats?
A scanned PDF, a price table in an image, a tutorial video with no transcript: to a retrieval system, all of that is noise or silence. Critical content must exist as structured text (clear headings, one topic per section, tables as tables). We cover how structure affects retrieval on our RAG and knowledge bases page.
6. Is there orphaned or zombie content?
Manuals for discontinued products, expired promotions, old versions of policies nobody deleted. To semantic search everything carries the same weight: a 2023 promotion can be the fragment most “similar” to today’s question. Before connecting anything, you need a purge: what gets deleted, what gets archived out of the assistant’s reach and what gets updated.
7. Is it clear what is public and what is internal?
In many companies, the public returns policy and the margin used to negotiate exceptions, the list price and the maximum authorised discount all live in the same space. If you connect the assistant “to everything”, sooner or later it will recite something internal to a customer. The public/internal separation has to exist at the source, with one simple rule: anything the assistant can read, it can say.
8. Does someone own each content area?
“The documentation belongs to everyone” means it belongs to no one. Each area (prices, policies, product, operations) needs an owner with a name who answers for its accuracy. When the assistant fails (it will), the question “whose content is this?” must have an immediate answer.
9. Is there an update process, or just good intentions?
Documentation doesn’t go stale out of laziness, but because updating isn’t part of any process: the price gets changed in the system and nobody has the task of changing it in the documentation. The rule we propose to our clients: no policy, price or product change is considered done until its document is updated. Without this, any assistant has an expiry date, as we explain in why chatbots fail in production.
10. Do you know which questions have no answer?
The last one and the most ignored: keep an explicit list of topics the assistant must not answer (legal, medical, cases that require a human) and of frequently asked questions that still have no document. The first defines your safety perimeter; the second is your documentation backlog, prioritised by real demand.
The right order: don’t wait for a 10/10
The wrong conclusion to draw from this checklist would be “first we fix all the documentation, then we launch the assistant”. It doesn’t work that way: without an assistant, there is no feedback on which documentation is missing or failing, and the “fix the documentation” project dies of abstraction within three weeks.
The order that works is the reverse: launch the assistant on a curated, reliable subset of the documentation (even a small one), with the perimeter tightly closed and with sources cited in every answer, and use what customers ask to prioritise what to document next. Each week’s failed conversations become the next week’s documentation to-do list. Within three months, the knowledge base is in better shape than ever — not because someone audited it in the abstract, but because every gap was flagged by a real question.
That cycle (failed conversations → documentation → better assistant) is exactly the recurring work included in our AI assistant, and the reason to distrust any chatbot project that starts with “upload your documents” instead of “let’s see what your customers ask”.