An AI Chat for Your Website That Answers Customers Fast and Reliably

TL;DR

• A visitor types a question in their own words and gets a short answer in a few seconds, with links to the exact pages it came from.
• The chat answers only from your own docs, blog and a hand-checked FAQ. If the answer is not there, it says so instead of guessing.
• How it works: your content is cut into small chunks, each chunk is turned into an embedding so it can be found by meaning, and for every question the best chunks go to an AI model that writes the answer from those chunks alone.
• It is cheap and the bill is bounded: about $0.0004 per question, a daily budget, and a $20/month hard cap.
• The second half of the post is the engineering: one small service, one model call per question, a search that put the right source in the top five for 92% of test questions, and source links the model cannot make up.

Contents

• What the visitor gets
• How it works, in plain words
• Under the hood: building it
• What a docs chat is for, and what it must never do
• The architecture: one service, one call, no database
• Retrieval: hybrid search, and why keywords get a smaller vote
• One call, and a prompt built so the question cannot take over
• Sources the model cannot invent
• What it costs, and how the bill stays bounded
• When something breaks, the chat disappears
• The knowledge base ships with every release
• Key takeaways

What the visitor gets

Someone is reading your product page and has one concrete question. Does it work without internet access? Does it run on our operating system? What does it cost for ten servers? The answer is in your docs somewhere, but finding it means leaving the page, guessing the right search term and reading three pages that are almost about it. Many visitors do not bother. They leave.

An AI chat on the website changes that moment. The visitor types the question the way they would ask a colleague, and a few seconds later there is a short answer, in their language, with links to the pages it was taken from (Figure 1). If your content does not cover the question, the chat says so plainly. It does not invent an answer, and it does not fall back on whatever the model happens to know about the topic in general.

Fast and reliable are the two promises here, and the rest of this post is about keeping both. Fast is the easy one: an answer arrives in one to five seconds. Reliable is where the thought goes. Every answer comes from your own reviewed content, every link points to a page the answer was really taken from, and a public text box can never turn into an open-ended bill.

How it works, in plain words

The technique has a name, retrieval-augmented generation, or RAG. The idea behind it is simple: before the AI writes anything, look up the few passages of your own content that answer the question, and let the AI write only from those. Figure 2 shows the whole process on one page.

Some of it happens ahead of time, before anyone asks anything:

1. Gather the knowledge. Collect what the chat is allowed to know: your documentation, your blog, and a short FAQ written and checked by a person. Nothing else goes in, which is what keeps the answers yours.
2. Split it into chunks. A whole page is too much to hand over for one question, and a single sentence is too little. So each page is cut into chunks of roughly one section, a few paragraphs each, and every chunk remembers which page and which heading it came from. That is what makes the source links possible later.
3. Turn every chunk into an embedding. More on this just below. In short, an embedding model turns each chunk into a list of numbers that describes what the chunk means.
4. Keep the result as a searchable library. All chunks with their numbers, ready to be searched. It is rebuilt automatically whenever the website changes, so the chat never knows less, or more, than the site.

Then, for every question:

5. The visitor asks. The question gets its own embedding, made the same way.
6. Find the relevant chunks. The chat looks for the chunks whose meaning is closest to the question, plus a classic keyword search for exact terms like product names and version numbers. The best eight chunks are kept.
7. The AI writes the answer. The question and those eight chunks go to a language model with a strict instruction: answer only from these chunks, and if they do not contain the answer, say so.
8. Check, then reply. Before anything reaches the visitor, the chat checks that every source the model cites is one of the chunks it was actually given, and turns those into links. Then the answer appears.

What is an embedding?

An embedding is a way to put text on a map of meaning. A special AI model reads a piece of text and returns a long list of numbers, its coordinates on that map. Texts that mean similar things end up close together, even when they use completely different words.

That is exactly what a website search usually lacks. A visitor asks "does it work without internet access?" while your docs say "air-gapped installation". The two share no word, so a keyword search misses the page. On the meaning map they sit side by side, so the chat finds it (Figure 3).

The map in the figure has two dimensions so it fits on a screen. A real embedding has about a thousand, but "close" means the same thing: the chat measures the distance between the question and every chunk and takes the nearest ones.

Under the hood: building it

That is the whole idea. The rest of this post is for whoever builds it: the design decisions, the numbers we measured, and the parts that make the chat reliable rather than merely impressive.

Years ago we spent a five-part series building a chat bot on Azure. Intents, utterances, a dialog tree: a lot of machinery to answer a fairly small set of questions. This October we built another one, a chat for a product documentation site, and the shape of the problem had changed completely. The language part is now the cheap part. What takes the thought is everything around it: what the model is allowed to know, how you stop it from inventing a source, and how you make sure a chat widget on a public page can never turn into an open-ended bill.

This is the pattern we landed on, with the numbers we measured along the way. None of it is specific to one site: it fits any docs, help-centre or product site with a few hundred to a few thousand pages.

What a docs chat is for, and what it must never do

The constraint to set before writing any code is strict: the chat answers only from your own knowledge base. That means the curated FAQ, the documentation and the blog. If the answer is not there, it says so. It does not fall back on what the model happens to know about the subject in general, because on a product site a confident answer that is not yours is worse than no answer.

Two smaller rules follow from that. The chat should speak the languages the site speaks (ours had to handle English and German). And when anything behind it is unavailable, the chat should simply not be there, rather than showing an error on a marketing page.

The architecture: one service, one call, no database

Figure 4 shows the whole system. The site itself is static and served from Cloudflare. A small Cloudflare Worker sits on /api/* of the same domain and forwards exactly the chat endpoints to a Go service running on a small k3s cluster, reached only through a Cloudflare tunnel. Because the API is on the same origin as the page, there is no CORS configuration at all.

A few decisions shape everything else:

• One model call per question. No tools, no agent loop. The service retrieves, calls Claude Haiku once with a structured-output schema, checks the result, and answers. Every question has the same, predictable cost and latency.
• No server-side conversation memory. The browser keeps the last three question/answer pairs in sessionStorage and sends them along, so "and on Windows?" still makes sense. The server stays stateless and stores no session at all.
• The knowledge base ships inside the image. More on that below, but it means there is no database to run, back up or secure.
• The widget answers the trivial inputs itself. Commands, aliases, keywords and typos of short inputs are matched in the browser at zero cost. Only real free-text questions go to the backend.

Retrieval: hybrid search, and why keywords get a smaller vote

The obvious first sketch puts the whole knowledge base into the prompt. It is a tempting shortcut for a small site, and we rejected it for two reasons: the cost of every question grows with every page you publish, and the model gets no signal about which pages actually answer the question, which makes honest source links much harder.

So the chat does retrieval-augmented generation (RAG). Pages are cut into chunks at each H2 heading, around 1,600 characters with a little overlap and the boilerplate dropped. On our test corpus that gave 2,680 chunks, each carrying its URL with the heading anchor, its title, language and type. Every chunk is embedded with bge-m3, a multilingual embedding model that runs comfortably on a consumer GPU (a GTX 1080 in our case).

At question time the service runs two searches over the same chunks in parallel: a vector search on the embedded question (about 45 ms at the median) and a classic BM25 keyword search. The two rankings are merged with reciprocal rank fusion. The interesting part is the weighting. Figure 5 shows what we measured on 24 representative questions.

Keywords alone found the right source in the top five for two thirds of the questions. Fusing both with equal weight helped, but the clear winner gave the vector ranking a weight of 1.0 and BM25 only 0.3: 92% at top-5, 96% at top-8. BM25 still earns its place. It catches exact product terms and version strings that embeddings blur, and it is the floor the chat falls back on when the embedding service is slow or down.

Two things worth leaving out:

• A vector database. 2,680 vectors of 1,024 dimensions are about 11 MB. The service loads them into memory and does brute-force cosine similarity, which takes well under a millisecond. An index structure or a hosted vector store would add a network hop, a bill and an operational dependency for a corpus that fits in RAM many times over.
• A reranker. We tested one over the top 30 candidates. On our GPU it added 4 seconds at the median, and it did not improve the top-8 hit rate at all. Leave it off until faster hardware and a measurable gain justify it.

One call, and a prompt built so the question cannot take over

The eight best chunks, about 3,000 input tokens, go into a single Messages call. The order of the prompt is deliberate, and Figure 6 follows one question through it.

The system block holds the rules: answer only from the supplied chunks; if they do not contain the answer, say so and set answered to false; answer in the language of the question; keep it short; cite chunk ids. Because the rules are byte-identical on every request, they are the one part of the prompt worth marking for caching. The chunks and the question change every time, so caching them would only cost the write premium.

Then come the chunks, each in a delimited block with a short id, its URL and heading. Then the recent conversation. Then the visitor's question, last. Both the question and the conversation are explicitly framed as data, not instructions, and the conversation is never sent as real assistant turns. The browser can be made to send anything as "history", including a fake earlier answer full of instructions. Placing it in a data block, rather than letting it speak in the model's own voice, is what keeps a forged turn from taking over.

Sources the model cannot invent

This is the step we would carry into any RAG system. The model is asked to cite chunk ids, never URLs. When the answer comes back, the server keeps a cited id only if that id was among the chunks retrieved for this request, maps it to the chunk's public URL from the index, and drops everything else. If no cited source survives, the answer is marked as not answered.

That is a stricter test than "the URL exists on the site". A plausible-looking link the model made up, or a real page it did not actually see, cannot reach the visitor. Every link in an answer points to text the model was shown.

The output itself comes back as structured JSON (answer, answered, sources) through the API's structured-output support rather than a prefilled response, and a refusal or a parse failure turns into a short, friendly fallback instead of a broken panel.

What it costs, and how the bill stays bounded

We measured real calls on Claude Haiku 5.5 before committing to it. A question with a warm prompt cache costs about four hundredths of a cent, and answers arrived in 1.4 to 4.6 seconds. Figure 7 puts that next to the two limits that bound the total.

A public text box that triggers a paid API call needs limits that do not depend on anyone remembering them. Layer them:

• Before any model call, a guard checks request size, question length (500 characters), the shape of the history, and a per-visitor rate limit. Take the visitor address from your edge (Cloudflare, in this build), not from a header the client could set, and use it only for counting: hash it, never log it, never store it.
• A daily budget in US dollars, computed from each response's actual token usage times the model's prices, so switching models cannot silently move the ceiling. When it is used up, the chat pauses until midnight UTC.
• A hard monthly cap enforced by the API provider on a workspace and key that are used for nothing else. Even a leaked key could not cost more than $20 a month, and it can be revoked without touching anything else.

When something breaks, the chat disappears

On page load the widget asks a cheap status endpoint whether the chat is ready. Only then does the launcher appear. If the service or the tunnel is down, or the day's budget is spent, the visitor sees no chat at all: no error, no layout shift. The site itself is served from the edge, so it stays up either way.

Degraded retrieval is a different case. If the embedding service is slow or unavailable, the chat keeps answering on keyword search alone. Hiding the widget is reserved for the case where it genuinely cannot answer.

The knowledge base ships with every release

Keep the chat in the same repository as the website. It turned out to be the most useful decision of all. On every website release, one pipeline builds the site, then builds the knowledge base from that very build, embeds only the chunks that are new or changed, bakes everything into the chat image, and deploys the site and the chat together.

Three properties fall out of that:

• Every source link exists on the live site by construction, because the index was generated from the exact pages being deployed. A page that is removed leaves the knowledge base in the same release.
• Embedding is incremental. Vectors are cached by a hash of the chunk text and the model id. On a typical release, only a few dozen of roughly 2,600 chunks are new, and embedding them takes seconds rather than the eight-plus minutes of a full run.
• Changing what the chat knows is a reviewed merge request. The hand-written FAQ entries are Markdown files in the repo, each stamped with who verified it and when. The chat repeats them nearly word for word, so they get a human reviewer like any other change. About ten to twenty-five minutes after the merge, the chat knows.

It is tempting to let the chat also query an internal knowledge base at runtime, since that is often where most of a team's knowledge lives. Don't. An internal store holds private material, and opening a public read path into it leaves you one wrong visibility flag away from a leak. Public answers should come from public pages, reviewed in the repo, and nothing else.

Key takeaways

• Start with what the chat must never do. "Only our own content, never a guess" decided the retrieval design, the prompt and the source check.
• Measure the fusion weights. Equal-weight hybrid search left nine points of hit rate on the table in our test.
• Small corpora do not need a vector database. Load the vectors into memory and keep the moving parts few.
• Let the model cite ids, and let code produce the links. It is the cheapest way to make invented sources impossible rather than unlikely.
• Put the cost limit somewhere you cannot forget it. A dedicated, capped API key is the backstop that holds even when your own counters reset.
• Ship the knowledge with the site. One pipeline, one release, and the chat can never point at a page that is not there.

If you are building something similar and want to compare notes, write to us at info@opensight.ch.