How PMs Decide RAG vs Fine-Tuning for Enterprise AI
For AI product managers at enterprises · Based on IBM LLM Customization Stack Framework
// TL;DR
Enterprise AI product managers face constant pressure to 'build a custom model' for legal, finance, or support use cases. The IBM LLM Customization Stack Framework gives you a defensible, ordered decision path: exhaust prompt/context engineering, RAG, and agent skills before approving any fine-tuning spend. It equips you to say no to premature training runs, articulate the exact bottleneck that justifies fine-tuning, and protect your roadmap from the Moving Target Problem — where a frontier release makes a custom model obsolete before it ships.
Why do product managers keep getting asked to fine-tune?
Stakeholders often equate 'the model doesn't know our domain' with 'we need a custom model.' As a PM, your job is to reframe that. In most enterprise cases — legal contract review, internal IT support, financial research — the gap is missing knowledge or missing procedure, not missing weights. The IBM LLM Customization Stack Framework gives you a structured way to redirect a fine-tuning request into a cheaper, faster, more durable solution.
How do you decide what customization method to approve?
Work down the stack in order and stop at the first layer that solves the problem:
1. Context engineering — Can a well-assembled prompt bundle (system prompt, relevant data, format rules, task instructions) solve it? Many use cases end here.
2. RAG — Is the knowledge proprietary, internal, or updated frequently (like weekly research reports)? Index it and retrieve at query time. No training run needed.
3. Agent skills — Is the gap procedural, like knowing your SQL schema or a compliance workflow? Package it into a skill file any model loads on demand.
4. Fine-tuning — Only if a concrete, measurable bottleneck remains: real-time latency, a need for a cheaper/smaller model, or programmatically gradable outputs.
Before approving a training run, require the team to name the exact bottleneck the first three layers cannot close. 'The model isn't accurate enough' is not specific enough.
How do you protect the roadmap from the Moving Target Problem?
Fine-tuning is a moving target: by the time a custom model ships, the next frontier release may have leapfrogged it. A legal AI preferred 97% of the time in 2023 was surpassed by seven general-purpose models by 2025. Mandate a domain-specific benchmark before and after any customization, and re-run it against every new frontier release — not just the model that existed at training time. This turns 'should we keep our custom model?' into a data-driven decision instead of a sunk-cost argument.
What hidden costs should you put in the business case?
Fine-tuning is not free. Beyond the training run, budget for labeled-data collection, evaluation cycles, regression testing to avoid degrading general capabilities, and ongoing maintenance as frontier models evolve. These are recurring costs. When you present a fine-tuning proposal to leadership, contrast this total cost of ownership against a RAG or agent-skills approach that ships faster and never goes obsolete when a new base model arrives.
What does a good approval decision look like in practice?
Suppose your firm wants an assistant over proprietary, weekly-updated research reports. That's proprietary, fresh knowledge — the stack says RAG, not fine-tuning. Index the reports, retrieve relevant documents at query time, and pass them into the prompt. You ship in weeks, updates require no retraining, and there's zero obsolescence risk. Reserve your fine-tuning budget for the one case that genuinely needs it — like a sub-500ms voice agent, where distillation to a small fast model is the legitimate answer.
As a PM, your leverage is in the decision, not the training run. Use the stack to convert vague 'build a custom model' requests into precise, defensible engineering choices.
Next step: Take your top three pending AI feature requests and run each through the four stack layers. For any that land on fine-tuning, write down the exact measurable bottleneck and draft the benchmark you'll use to validate — and periodically re-validate — the investment.
// FREQUENTLY ASKED QUESTIONS
How do I push back when leadership demands a custom fine-tuned model?
Reframe the request around the stack. Ask what specific bottleneck fine-tuning would solve, then show that context engineering, RAG, or agent skills likely close the gap faster and cheaper. Present the hidden costs — labeled data, evaluation, regression testing, maintenance — and the Moving Target risk. Data from a benchmark comparing approaches makes the case far better than opinion.
What should I require before approving a fine-tuning budget?
Require three things: a precise, measurable bottleneck the non-weight-touching stack cannot solve; a domain-specific benchmark on your actual use case; and a plan to re-run that benchmark against future frontier releases. If the team can't articulate the exact bottleneck, the request goes back down the stack to RAG or agent skills first.
How do I estimate the total cost of ownership for a custom model?
Include the training run plus recurring costs: labeled-data collection, evaluation cycles, regression testing to protect general capabilities, and re-benchmarking or re-training each time a frontier model ships. Compare this ongoing TCO against a RAG or agent-skills solution, which typically ships faster and avoids obsolescence entirely.