RAG for Business: When You Need It, When You Don't

Every few weeks someone in Burnaby or New Westminster asks us the same question: "Do we need RAG?" They heard the acronym from a vendor and nobody explained it. So here it is in plain language, including the part most vendors skip, which is that a lot of the time you don't need it.

What RAG actually is

RAG stands for retrieval augmented generation. Underneath the acronym it's a two step move. First, search your own files for the few passages that relate to the question someone asked. Second, paste those passages into the AI's prompt and tell it to answer from those only.

That's the whole idea. The model doesn't learn anything and nothing gets retrained. It just gets handed the right page before it answers.

The term comes from a 2020 NeurIPS paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Their framing is still the cleanest one available. A model has parametric memory, meaning facts baked into its weights, and RAG bolts on non-parametric memory, meaning an index it can look things up in. The paper reports that RAG models "generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline."

Two practical things follow. You change what the AI knows by editing a document instead of retraining a model. And every answer can carry a citation.

The one question that decides it

Ask yourself where the correct answer lives.

If it lives in your head, or in a policy short enough to write in a paragraph, a good prompt is enough. Put the rules in the system prompt and move on.

If it lives scattered across hundreds or thousands of documents that keep changing, and nobody can predict which document a given question will need, you're in retrieval territory.

Microsoft's Azure AI Search team frames it as a token problem. Their RAG overview, updated August 4, 2026, notes that "LLMs accept limited token inputs" and that a retrieval system has to return concise, highly relevant results rather than dumping whole documents into the prompt.

But the context windows are huge now

Fair objection. If a model accepts a very long input, why not paste the entire manual every time?

Because long input does not mean evenly read. In Lost in the Middle: How Language Models Use Long Contexts, published in Transactions of the Association for Computational Linguistics, Nelson Liu and co-authors found that performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models."

Models have improved since that study came out. The planning lesson still holds: a handful of well chosen passages beats a giant pile. Retrieval is a relevance problem before it's a storage problem.

When a good prompt is enough

Skip RAG, at least for now, when any of these describe you:

• Your knowledge fits on two pages. Hours, service area, pricing tiers, booking rules, cancellation policy. That belongs in the prompt.
• The content barely changes. If it moves twice a year, edit the prompt twice a year.
• The task is transformation, not lookup. Rewriting a quote, summarizing a call, drafting a follow up. There is nothing to retrieve.
• The same reference text goes out on every request. Caching covers this. Anthropic's prompt caching documentation lists cache read tokens at 0.1 times the base input token price, with a default cache lifetime of five minutes, so repeating a fixed block is cheap.
• Your documents are a mess. Retrieval over a disorganized shared drive gives you disorganized answers, faster.

What RAG costs that a prompt doesn't

A retrieval system is not one thing. It's document chunking, an embedding step, a vector index, a refresh job for when files change, permission handling so the wrong person can't pull the wrong file, and an evaluation set so you can tell when a change made answers worse.

Security is the piece small businesses underestimate. Microsoft's page is blunt: "Opening private content to LLMs requires granular access control. Users and agents must only retrieve authorized content." A vector index does not inherit your folder permissions by default. Somebody has to build that.

Here's an illustrative comparison, not a client result. If your AI receptionist handles 300 calls a month and twelve questions cover most of them, writing twelve good answers into a prompt takes an afternoon. A retrieval pipeline over the same content takes weeks and then needs an owner. The prompt wins until those twelve questions become two hundred.

The underrated reason to build it: citations

When RAG earns its keep, it's often less about raw accuracy and more about being checkable. Anthropic's search results documentation describes content blocks that "let Claude cite your own content the same way it cites web search results: each citation carries the source and title you provided," and notes that no beta header is required because they're part of the standard Messages API.

That matters for a contractor quoting warranty terms or a clinic answering coverage questions. Staff can open the citation and confirm before anything reaches a customer.

Privacy, which matters more than the architecture

Whatever you index, you're making a copy of it. The Office of the Privacy Commissioner of Canada, together with provincial privacy commissioners, published Principles for responsible, trustworthy and privacy-protective generative AI technologies in December 2023. One principle is to "avoid indiscriminate collection of personal information based on assertions about the breadth of potential purposes." Another asks organizations to ensure that "procedures exist for individuals to access and correct any information collected about them."

Read that practically: index the handbook and the service docs, not the entire client folder. If personal information has to be in there, know in advance how you'd find it and correct it.

Where this doesn't apply

RAG does not fix bad source material. If two documents contradict each other, retrieval finds both and the AI picks one. Cleaning up your content is the prerequisite, not the bonus round.

If the answer is a number sitting in a database, don't embed it. Give the AI a query tool instead. RAG is for prose. Same goes for live data such as inventory counts or today's schedule, which wants an API call, not an index refreshed overnight.

And the tooling is still moving. Microsoft now steers new builds toward agentic retrieval, recommending the classic pattern when you need generally available features only, or fine grained control over the query pipeline. The recommended shape shifted inside a couple of years, so whatever you build, build it so the retrieval layer can be swapped without rewriting everything around it.

Some perspective on how early this is. Statistics Canada reported that in the second quarter of 2026, 19.2% of Canadian businesses had used AI to produce goods or deliver services over the previous twelve months, tripled from 6.1% in the second quarter of 2024. Among businesses with 1 to 4 employees the figure was 19.9%, and virtual agents or chat bots ranked third among applications used, at 28.2%. Most Metro Vancouver shops are still on their first AI project. Starting with a sharp prompt is a legitimate starting point, not a failure of nerve.

Sources

• Lewis, P., Perez, E., Piktus, A., et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS, 2020. arxiv.org/abs/2005.11401
• Liu, N. F., Lin, K., Hewitt, J., et al. "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the Association for Computational Linguistics, 2023. arxiv.org/abs/2307.03172
• Microsoft. "RAG and Generative AI, Azure AI Search." Microsoft Learn, updated August 4, 2026. learn.microsoft.com
• Anthropic. "Search results." Claude Platform documentation, 2026. platform.claude.com
• Anthropic. "Prompt caching." Claude Platform documentation, 2026. platform.claude.com
• Office of the Privacy Commissioner of Canada and provincial privacy commissioners. "Principles for responsible, trustworthy and privacy-protective generative AI technologies." December 2023. priv.gc.ca
• Statistics Canada. "Analysis on artificial intelligence use by businesses in Canada, second quarter of 2026." Released June 11, 2026. statcan.gc.ca

If you're not sure which side of the line you're on, that's a short conversation, not a project. Autana Solutions is based in Burnaby and works with businesses across Metro Vancouver. Book a free call, show us your actual documents, and we'll tell you honestly whether you need retrieval or just a better prompt.