Articles/Can Your Firm Train Its Own AI Model? Pre-Training, Fine-Tuning, and RAG, Explained
Can Your Firm Train Its Own AI Model? Pre-Training, Fine-Tuning, and RAG, Explained
Pre-training and fine-tuning are both called training — one takes a frontier lab, the other runs on a Mac mini. Firms announcing they'll train their own model are usually describing the one they can't afford.
“We’re going to train our own model”
This sentence is showing up in AEC strategy decks. A firm has decades of drawings, specs, and project data, and someone concludes the obvious next move is to train a foundation model on it — a proprietary “firm brain.” It sounds ambitious but tractable — a big IT project, like standing up a server farm.
It isn’t. Training a foundation model means starting from random numbers and running trillions of tokens through tens of thousands of GPUs until general language ability emerges. The compute bill alone runs into the hundreds of millions at the frontier — before energy, infrastructure, and the research team you need to keep the run from collapsing. The labs that do this spend billions a year to stay in the game. A handful of them can do it at the level you’d actually want to use. Your firm’s archive, however good, is a rounding error against the data required.
But here’s what makes this confusing rather than just humbling: the thing your firm can do to a model is also called training. Same word, different phase, wildly different price. Firms that don’t know the vocabulary either announce the impossible — or overcorrect and conclude models can’t be changed at all. Both are wrong.
Two phases, one word
Pre-training is the frontier-lab phase described above. When someone says “training a foundation model is out of reach,” this is what they mean, and they’re right.
Fine-tuning is everything after. You take an already-pre-trained open model — Llama, Mistral, Qwen — and continue updating its weights on your own data to shift its behavior in a specific direction. Feeding it curated input/output pairs is called supervised fine-tuning. And it is genuinely training in the technical sense: gradient descent, weight updates, the same math as pre-training. It just starts from a good checkpoint instead of noise, and needs a few thousand examples instead of a few trillion tokens.
The industry keeps these phases straight with different words. Strategy decks don’t — which is how “we’ll train our own model” gets pitched at pre-training scale when the thing actually being described is a fine-tune.
Why it fits on a Mac mini
The technique that makes desktop fine-tuning practical is called LoRA — low-rank adaptation (its memory-squeezed cousin is QLoRA). Instead of updating all of a model’s billions of weights, LoRA freezes them and trains small adapter matrices alongside — often well under 1% of the parameter count.
The intuition: you’re not reshaping the model’s whole understanding of language. You’re teaching it a narrow set of adjustments — a few directions, in one domain. How your firm writes a basis-of-design narrative. How your spec sections are structured. The exact format of your drawing-review comments.
The output is an adapter file measured in megabytes, not a new model measured in hundreds of gigabytes. Train it in hours on a well-specced Mac mini — for the 7-to-13-billion-parameter open models this technique targets — load it on top of the base model, swap it out when it’s stale.
One honest caveat: the GPU time is hours, but the real cost is upstream. Someone who knows your documents has to curate those few thousand examples, and that’s weeks, not hours.
The line that actually matters
Here’s the dividing line worth remembering, because vendors blur it constantly:
If a gradient update touched the weights, it’s training. If not, it isn’t.
Pre-training and fine-tuning both pass this test. Almost everything else marketed as “training” doesn’t — starting with the most common case.
Is RAG training?
No. And this is where the vocabulary gets blurred most often, because “we trained it on your data” is exactly how retrieval products get sold.
RAG — retrieval-augmented generation — works like this: your documents are converted into embeddings (numerical fingerprints of their meaning), stored in a database, and when someone asks a question, the system fetches the most relevant passages and pastes them into the prompt alongside the question. The model reads them the way it reads anything else in a prompt, answers, and forgets them.
Nothing in the model changed. No gradient, no weight update, no adapter. Delete the database tomorrow and the model behaves as if it never saw your documents — because in the training sense, it never did. RAG doesn’t teach the model your archive; it hands the model the right page of your archive at the moment of the question.
Two nearby claims that blur the same line:
- “It gets smarter the more you use it.” Memory features in chat products are retrieval too — past conversations stored and re-injected into context. The weights don’t move.
- “It embedded our documents, so it learned them.” The embedding model was trained — by someone else, long before your documents arrived. Running your documents through it is inference, not training.
None of this makes RAG the lesser option. For factual knowledge it’s usually the better one: it updates instantly (edit the document), it can cite its sources, and it can’t confidently misremember the way a fine-tuned model can. The practical split is that fine-tuning is good at behavior and unreliable at facts, while RAG is the reverse. That’s why the default stack for a firm is retrieval for knowledge, skills for procedure — and fine-tuning only for the narrow cases below.
| What changes | Cost & effort | Undo it? | |
|---|---|---|---|
| Pre-training | Everything, from scratch | Hundreds of millions+, a lab | N/A |
| Fine-tuning (LoRA) | A small adapter on top of frozen weights | A desktop, hours of compute, weeks of data prep | Delete the adapter |
| RAG / skills / prompts | Nothing in the model | Minutes to edit a file | Delete the database |
Should your firm fine-tune?
Honestly: probably not first. We’ve made this argument before — most of what a firm needs is skills, context, and access to its own documents, not a modified model. A SKILL.md file is inspectable, editable, and works with frontier models that are already better than anything you can run locally. Fine-tuning generally means a smaller open model — vendors do offer hosted fine-tuning of bigger models, but then your data lives with the vendor, which defeats the most common reason firms want this.
Fine-tuning earns its place in narrower cases: when you need consistent behavior that’s hard to prompt for (a very specific output format, a house style across thousands of generations), when data can’t leave your infrastructure at all, or when you’re running the same narrow task at volume and a small specialized model beats renting a big general one.
So when the strategy deck says “we’re going to train our own model,” the right response isn’t no — it’s a translation. What the firm actually wants is a fine-tune, or more often just skills and retrieval over its own documents. The frontier-lab barrier applies to making foundation models, not to changing them. Knowing which word you mean is the difference between a real roadmap and a slide.