Fine Tuning vs RAG: What Each Actually Costs in 2026

Someone asks us most weeks whether they should fine-tune a model on their own business data. Usually the answer is no. Fine-tuning isn't bad. It's just the most expensive way to fix a problem that a sharper prompt or a document lookup would have solved for a fraction of the money.

Adoption is no longer the edge case it was. Statistics Canada reports that in the second quarter of 2026, 19.2% of Canadian businesses used AI to produce goods or deliver services, roughly triple the 6.1% recorded in the second quarter of 2024, with urban businesses at 21.0% against 9.9% rural (Statistics Canada, 2026). So the question for most shops around Metro Vancouver is no longer whether to use this stuff. It's which version you're paying for.

The three options in plain terms

Prompting means putting your rules, tone and examples directly into the request. No training, no infrastructure.

RAG, or retrieval augmented generation, means searching your own documents first and pasting the relevant pieces into the prompt before the model answers. OpenAI's guide calls it "the process of Retrieving content to Augment your LLM's prompt before Generating an answer," and points to it when "the model lacks contextual knowledge because it wasn't in its training set, its knowledge is out of date, or it requires knowledge of proprietary information" (OpenAI, Optimizing LLM accuracy).

Fine-tuning means retraining the model's weights on your labelled examples so it behaves a certain way without being told each time.

They fix different problems. RAG supplies facts. Fine-tuning shapes behaviour. Mixing those up is where budgets quietly disappear.

Fine tuning vs RAG: what the research actually found

Microsoft researchers compared unsupervised fine-tuning against RAG on knowledge-heavy tasks and found that "while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge." They also note that "LLMs struggle to learn new factual information through unsupervised fine-tuning" (Ovadia et al., 2023).

A 2024 study ran twelve models against exactly the kind of obscure facts a single business owns. Fine-tuning did help. It just lost: "RAG surpasses FT by a large margin particularly for least popular factual knowledge" (Soudani et al., 2024).

If you want the AI to know your price list, your service area, your warranty terms or which brands you stock, fine-tuning is both the weaker tool and the pricier one.

The price tags, as of August 2026

Here's what the published rate cards say right now. Check them again before you budget, because they move.

• Prompting costs tokens and nothing else. Claude Sonnet 5 is listed at $2 per million input tokens and $10 per million output (Anthropic pricing).
• Caching cuts the repeated part hard. Anthropic prices cache reads at 0.1x the base input rate, so a cached Sonnet 5 prompt reads at $0.20 per million tokens. A five minute cache write costs 1.25x, which means caching "pays off after one cache read" (Anthropic prompt caching).
• Batch work gets 50% off both input and output tokens through Anthropic's Batch API, if the job can wait.
• RAG adds a search index, embedding costs and retrieved chunks in every prompt. At small business volume that's usually tens of dollars a month on top of tokens, plus the work of keeping documents clean.
• Fine-tuning training is cheap and one time. OpenAI lists gpt-4.1 training at $25.00 per 1M training tokens (OpenAI pricing). Five hundred examples of about 1,000 tokens each over three epochs is 1.5M training tokens, so roughly $37. That figure is illustrative, but the rate is published.
• Fine-tuned inference is not cheap, and it never ends. Base gpt-4.1 lists at $2.00 in and $8.00 out per million tokens. The fine-tuned version of the same model lists at $3.00 in and $12.00 out. On gpt-4.1-mini the gap is exactly double: $0.40 and $1.60 becomes $0.80 and $3.20.

The number most people miss

Say you run an after-hours booking agent that carries 4,000 tokens of standing context on every conversation: services, hours, policies, booking rules. At 1,000 conversations a day that's 4 million input tokens daily. On Claude Sonnet 5 at $2 per million, uncached, you're at about $8 a day on the input line alone, near $240 a month before a single word of output.

Turn on prompt caching and those same tokens read at $0.20 per million. That's about $0.80 a day, call it $25 a month once you count cache writes. Same model, same answers, same context, roughly 90% off the input line. These figures are illustrative, but the multiplier comes straight from Anthropic's published rates.

Now put fine-tuning next to it. The training run was $37. The 50% inference premium, though, applies to every token you ever send. And the day your prices change or you add a location, you retrain. With retrieval you edit a document and you're done.

When fine-tuning genuinely wins

Fine-tuning earns its money when the problem is behaviour, not knowledge. A rigid output format your other systems have to parse. A voice you need held consistently across ten thousand messages. A classification job where you want a small cheap model to match a big model's judgement, which is the case where fine-tuning actively saves money, since a tuned mini model at $0.80 per million input tokens costs far less to run than a frontier model at $5.

OpenAI's own guidance is blunt about sequencing. "Prompt engineering is typically the best place to start. It is often the only method needed for use cases like summarization, translation, and code generation." On dataset size: "Quality of training data is more important than quantity... Start with 50+ examples." And the line worth taping to the wall: "squeeze as much accuracy out of basic methods as you can before reaching for more complex RAG or fine-tuning."

Where this doesn't apply

A few honest limits.

If your documents are a mess, RAG inherits the mess. Retrieval quality caps answer quality, and cleaning up twelve years of contradictory PDFs is the real project, not the AI.

If you handle thirty conversations a day, none of this arithmetic matters. The gap between $8 and $0.80 a day doesn't fund a project. Write a good prompt, run it for a quarter, revisit when volume justifies the build.

The sources don't fully agree either. Soudani et al. found fine-tuning "boosts the performance across entities of varying popularity," so it isn't useless for facts, just outmatched. OpenAI also recommends that if you already run RAG, you "fine-tune the model with RAG examples in it so it isn't learning how to use the context zero-shot." At high accuracy targets these are complements, not rivals.

Prices are the least stable thing here. Anthropic's Sonnet 5 rate was announced as introductory with an increase scheduled for September 1, 2026, and that increase was then cancelled and the introductory price made standard. Anything you read about model pricing, including this post, needs re-checking against the vendor page.

Privacy cuts the same direction. Canadian privacy regulators expect organizations to "limit the collection, use, and disclosure of personal information to only what is needed to fulfill the explicitly specified, appropriate identified purpose" and to "recognize that they are responsible for compliance with privacy legislation" (OPC and provincial commissioners, 2023). Fine-tuning bakes customer records into weights you cannot easily unbake. A retrieval index lets you delete a file. For a Burnaby clinic or a law office holding client information, that difference matters more than the token math.

The order we actually use

Prompt first, with real examples, and measure it. Add caching once the standing context is stable. Add retrieval when the failures are knowledge gaps rather than behaviour. Fine-tune only when behaviour is still inconsistent and your volume makes the permanent inference premium worth it. Most of the businesses we talk to stop at step two or three, and their bill reflects it.

Sources

• OpenAI. "Optimizing LLM accuracy," OpenAI API documentation, accessed August 2026. https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy
• OpenAI. "Pricing," OpenAI API documentation, accessed August 2026. https://developers.openai.com/api/docs/pricing
• Anthropic. "Pricing," Claude Platform documentation, accessed August 2026. https://platform.claude.com/docs/en/about-claude/pricing
• Anthropic. "Prompt caching," Claude Platform documentation, accessed August 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
• Ovadia, O., Brief, M., Mishaeli, M., Elisha, O. "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs," arXiv:2312.05934, 2023. https://arxiv.org/abs/2312.05934
• Soudani, H., Kanoulas, E., Hasibi, F. "Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge," arXiv:2403.01432, 2024. https://arxiv.org/abs/2403.01432
• Statistics Canada. "Analysis on artificial intelligence use by businesses in Canada, second quarter of 2026," 2026. https://www150.statcan.gc.ca/n1/pub/11-621-m/11-621-m2026010-eng.htm
• Office of the Privacy Commissioner of Canada and federal, provincial and territorial privacy authorities. "Principles for responsible, trustworthy and privacy-protective generative AI technologies," 2023. https://www.priv.gc.ca/en/privacy-topics/technology/artificial-intelligence/gd_principles_ai/

If you're weighing these options for your own shop, we're happy to look at the actual numbers with you and tell you honestly if a prompt will do. Book a free call with Autana Solutions and we'll map your use case to the cheapest thing that works.