Retrieval-Augmented Generation, Explained for Product Teams

"Just add RAG" has become shorthand for "make the AI feature know about our data," which undersells how many decisions sit behind that phrase. Retrieval-augmented generation is a simple idea with a lot of ways to implement it badly.

The idea in one paragraph

An LLM only knows what was in its training data and whatever you put in its prompt. RAG closes that gap: instead of asking the model a question directly, you first search your own content for the passages most relevant to the question, then hand the model those passages alongside the question and ask it to answer using only that context. The model isn't recalling your data from memory — it's reading it, every time, like a very fast research assistant.

Why not just fine-tune the model instead?

Fine-tuning bakes patterns into model weights; it's good for teaching a model a style, a format, or a narrow skill. It's a poor fit for "know about our product docs," because:

• Your docs change constantly, and fine-tuning is expensive to repeat on every update.
• Fine-tuned models still hallucinate — they don't reliably cite where an answer came from.
• You can't easily audit or update a single fact after the fact.

RAG sidesteps all three: update the source documents, and the next query picks up the change immediately, because retrieval happens at query time, not training time.

The part everyone underestimates: chunking

Before anything can be retrieved, your content has to be split into chunks small enough to embed and retrieve individually. Chunk too large, and you retrieve a lot of irrelevant text alongside the useful sentence — diluting the context and wasting tokens. Chunk too small, and you lose the surrounding context a passage needs to make sense on its own.

There's no universal chunk size. A FAQ page chunks naturally by question. A long-form technical guide needs chunking that respects section boundaries (don't split a code example from the paragraph explaining it). This is usually where a RAG system's quality is actually won or lost — not in which vector database you pick.

Embeddings and retrieval

Each chunk gets converted into an embedding — a vector representation of its meaning — and stored in a vector database. At query time, the user's question is embedded the same way, and the database returns the chunks whose vectors are closest to it.

const queryEmbedding = await embed(userQuestion);
const matches = await vectorDb.query({
  vector: queryEmbedding,
  topK: 5,
  filter: { tenantId },
});

Two details matter more than people expect:

• Hybrid search. Pure vector similarity misses exact keyword matches (a product SKU, an error code). Combining vector search with keyword search catches both.
• Metadata filtering. If your content spans multiple tenants, products, or access levels, filter at the database query, not after retrieval — a retrieved chunk that belongs to a different customer is a data leak, not just a bad answer.

Generation: give the model an out

The final prompt should give the model the retrieved chunks, the question, and — critically — explicit permission to say it doesn't know:

Answer using only the context below. If the context doesn't contain
the answer, say you don't have that information — do not guess.

Context:
{{retrieved_chunks}}

Question: {{user_question}}

Without that instruction, models will confidently answer from general knowledge when your retrieved context is thin, which is exactly the hallucination RAG was supposed to prevent. Validate the response shape with a schema (we use Zod for this) so a malformed or empty answer fails safely instead of rendering broken output in your UI.

When RAG isn't the right tool

If the "knowledge" you need is small enough to fit entirely in a prompt — a product's pricing table, a short policy document — just put it in the system prompt and skip the retrieval infrastructure entirely. RAG earns its complexity when your content is too large to fit in context, changes often, or needs to be scoped per user or tenant. Reach for it because the problem demands it, not because it's the default AI-feature pattern.

We build this end to end — chunking strategy, retrieval pipeline, and the generation layer around it — as part of AI development engagements, and if you want to build this kind of system hands-on, it's a core module in the AI development course.