IBM LLM Customization Stack Framework

Last updated: 24 July 2026

Given any AI deployment scenario, determine the optimal customization path — from prompt engineering through RAG, agent skills, and fine-tuning — without wasting resources on training that frontier models may leapfrog overnight.

// TL;DR

The IBM LLM Customization Stack Framework is a decision method for making a general-purpose LLM behave like a specialist without wasting resources on unnecessary training. It orders customization techniques from cheapest to most expensive: prompt/context engineering, then RAG, then agent skills, and only then fine-tuning (LoRA, distillation, or RFT). Use it whenever a team must decide how to specialize an AI for a domain like legal, finance, or support — and needs to avoid the 'Moving Target Problem,' where a frontier release leapfrogs a custom model before it even ships.

Framework

// When should you use the IBM LLM Customization Stack Framework?

Use this skill whenever a team or individual needs to decide how to make a general-purpose LLM behave like a specialist — e.g., for legal, financial, support, or domain-specific tasks — and must choose between weight-touching and non-weight-touching customization methods.

// What do you need before applying the Customization Stack?

Use case descriptionrequired
What task or domain does the AI need to specialize in? (e.g., legal contract review, internal IT support, real-time voice response)
Latency requirementsrequired
Does the application require real-time or near-real-time responses (e.g., voice agent), or is latency flexible?
Data profilerequired
Is the required knowledge publicly available on the internet, or is it proprietary/internal/fresh data not in any base model's training set?
Output gradability
Can the model's outputs be programmatically graded — i.e., is there a definitive right answer?
Resource constraints
What is the team's capacity for ongoing model maintenance, training runs, and evaluation cycles?

// What principles guide LLM customization decisions?

The Moving Target Problem

Fine-tuning a custom model is a moving target: by the time the fine-tuned model ships, the next frontier release may have leapfrogged it already. Always weigh the shelf life of a custom model against the pace of frontier improvement before committing to a training run.

Don't Bake What You Can Pass

If a model can read 500 pages of documents directly in its prompt via an expanded context window, there is no need to bake those documents into the weights at all — just pass the contextually relevant material at query time.

The Customization Stack (Non-Weight-Touching First)

Fine-tuning is not the only path to specialization. There is a whole stack of customization techniques — prompt/context engineering, RAG, and agent skills — that work without ever touching model weights. Exhaust this stack before reaching for fine-tuning.

Fine-Tuning Is Not Free

Beyond the training run itself, fine-tuning carries hidden costs: collecting labeled examples, evaluating results, avoiding regressions, and maintaining the custom model as frontier models move on. These costs must be explicitly justified.

Reasoning at Inference Time

Reasoning models do extended thinking at inference time, working through a problem step by step before answering. Specialization can therefore come from how hard the model thinks at the moment of the question, not just from how it was trained months earlier.

// How do you apply the Customization Stack step by step?

1
Start with a base model and apply prompt and context engineering
A good prompt is a carefully assembled bundle of context: the system prompt, relevant data, format guidelines, and task instructions all packaged together. This is 'context engineering.' Treat this as the mandatory first layer — many use cases are solved here alone. Do not skip to weight-touching methods until this is verified insufficient.
2
Assess whether knowledge is fresh or proprietary — if so, add RAG
RAG (Retrieval-Augmented Generation) retrieves documents at query time and feeds them into the prompt, rather than training them into the weights. Use RAG when the required knowledge is not on the public internet (internal support tickets, private contracts, real-time data). RAG solves the 'stale weights' problem without a training run.
3
Assess whether procedural know-how is missing — if so, add agent skills
Agent skills are folders of files (e.g., markdown files) that package up procedural knowledge: how to do something, and the tools to use to do it. The model loads them on demand when it sees a task that calls for them. Example: instead of fine-tuning a model to know how to write SQL against a specific schema, an SQL agent skill tells any general-purpose model exactly what to do. Build skills for repeatable procedures before considering fine-tuning.
4
Identify whether a specific bottleneck remains that the stack cannot solve
Only reach for fine-tuning if there is a concrete, measurable gap that prompt engineering, RAG, and agent skills cannot close. Articulate the bottleneck precisely before proceeding. Valid bottlenecks include: real-time latency requirements, need for a smaller/cheaper distilled model, or outputs that can be programmatically graded for reinforcement fine-tuning.
5
If fine-tuning is justified, select the appropriate fine-tuning method
Default to LoRA (Low-Rank Adaptation): train a small adapter that sits on top of the base model with most original weights locked. This is the dominant production fine-tuning method today. If the goal is a smaller, cheaper model, use distillation: generate high-quality outputs from a large 'teacher' model (including reasoning traces), then fine-tune a smaller 'student' model on those outputs. If outputs are programmatically gradable, consider RFT (Reinforcement Fine-Tuning): use a prompt dataset plus a grader; the model samples candidate answers, the grader scores them, and training updates the model to make high-scoring answers more likely. RFT only works when there is a definitive right answer.
6
Build a benchmark before and after to validate the investment
Any fine-tuning effort must include a domain-specific benchmark that measures task performance on the actual use case. Test the custom model against current frontier models — not just the frontier model at the time of training. The legal AI case study shows that a model preferred 97% of the time in 2023 was surpassed by seven general-purpose models by 2025. Benchmarks must be re-run against new frontier releases regularly.

// What are real examples of the Customization Stack in action?

A financial services firm wants to build an AI assistant that answers questions about its proprietary internal research reports, which are updated weekly and never published publicly.

The knowledge is proprietary and fresh — the Customization Stack says add RAG. Index the internal reports into a retrieval system; at query time, retrieve the relevant documents and pass them into the prompt. No fine-tuning needed. Revisit only if retrieval quality is provably insufficient after context engineering optimizations.

A software company wants an AI coding assistant that always follows its internal database schema and coding conventions when writing SQL queries.

This is a procedural know-how gap — the Customization Stack says add agent skills. Package the schema, conventions, and step-by-step SQL generation procedure into an agent skill (a set of markdown/config files). Any general-purpose model can load this skill on demand. No weight-touching required.

A customer support company needs a voice agent that must respond within 500 milliseconds; frontier reasoning models are too slow due to their extended thinking time.

Real-time latency is the bottleneck — a valid fine-tuning trigger. Use distillation: generate high-quality support responses from a large frontier teacher model, then fine-tune a small, fast student model on those outputs. Deploy the small fine-tuned model for voice. Re-evaluate when smaller frontier models improve further.

An edtech company wants an AI math tutor that scores student answers and gets progressively better at identifying correct solution paths.

Outputs are programmatically gradable (math has definitive right answers) — a valid use case for RFT (Reinforcement Fine-Tuning). Build a prompt dataset of math problems plus a grader that scores answers. Run RFT so the model learns to make high-scoring solution paths more likely. Confirm the bottleneck cannot be solved by prompt/context engineering first.

// What mistakes should you avoid when customizing an LLM?

Reaching for fine-tuning as a first instinct rather than a last resort — exhaust the non-weight-touching Customization Stack first (prompt/context engineering → RAG → agent skills).
Ignoring the Moving Target Problem: training a custom model that a new frontier release leapfrogs before it even ships.
Benchmarking a fine-tuned model only against the frontier model that existed at the time of training, not against current frontier releases.
Underestimating the hidden costs of fine-tuning: labeled data collection, regression testing, evaluation cycles, and ongoing maintenance as frontier models evolve.
Using RFT (Reinforcement Fine-Tuning) on tasks where outputs cannot be programmatically graded — RFT only works when there is a definitive right answer.
Assuming expanded context windows don't change the calculus — if a model can read 500 pages directly in a prompt, baking those pages into weights is wasteful.
Conflating 'the model doesn't know our domain' with a fine-tuning problem — it is usually a RAG or context engineering problem.

// What are the key terms in LLM customization?

Base Model: An off-the-shelf LLM trained on a massive, broadly scraped dataset whose general knowledge is baked into its weights. The starting point for all customization decisions.
Fine-Tuning: Continuing the training of a base model on a focused, domain-specific dataset, producing a new model whose weights incorporate both the original general knowledge and the specialized data.
The Customization Stack: The ordered set of techniques for specializing a general model — prompt/context engineering, RAG, and agent skills — all of which work without touching model weights. Fine-tuning sits at the bottom of the stack and should only be reached if the rest cannot solve the bottleneck.
Context Engineering: The practice of assembling a carefully structured prompt bundle — system prompt, relevant data, format guidelines, and task instructions — to make a general model behave like a specialist without any training.
RAG (Retrieval-Augmented Generation): A technique where an application retrieves relevant documents at query time and feeds them into the prompt, rather than training the documents into the model's weights. Solves the problem of proprietary or fresh knowledge.
Agent Skills: Folders of files (e.g., markdown files) that package up procedural knowledge — how to do something and which tools to use — which any general-purpose model can load on demand when it encounters a matching task.
LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning method that trains a small adapter sitting on top of a base model, leaving most of the original weights locked. The dominant production fine-tuning approach today.
Distillation: A fine-tuning strategy where a large frontier 'teacher' model generates high-quality outputs (including reasoning traces), and a smaller 'student' model is fine-tuned on those outputs to inherit the teacher's capability at lower cost and latency.
RFT (Reinforcement Fine-Tuning): A training approach using a prompt dataset plus a programmatic grader: the model samples candidate answers, the grader scores them, and training updates the model to make high-scoring answers more likely. Only applicable when a definitive right answer exists.
Reasoning Models: LLMs that perform extended thinking at inference time — working through a problem step by step before answering — deriving specialization from how hard they think at the moment of the question rather than purely from training.
The Moving Target Problem: The risk that by the time a fine-tuned custom model ships, a new frontier release has already surpassed it, making the training investment obsolete.
Benchmark: A domain-specific evaluation suite that measures how effectively a model performs particular tasks in the target domain, used to validate whether customization efforts actually improve on frontier baselines.

// FREQUENTLY ASKED QUESTIONS

What is the IBM LLM Customization Stack Framework?

It's an ordered decision framework for specializing a general-purpose LLM without wasting resources. You exhaust non-weight-touching methods first — prompt/context engineering, RAG, then agent skills — and only reach for fine-tuning (LoRA, distillation, or RFT) if a concrete, measurable bottleneck remains. The core idea: don't train weights when passing context, retrieving documents, or loading a skill file solves the problem.

What is the difference between RAG and fine-tuning?

RAG retrieves relevant documents at query time and feeds them into the prompt, so knowledge stays fresh and no training run is needed. Fine-tuning bakes knowledge into the model's weights through additional training. Use RAG for proprietary or frequently-updated knowledge; use fine-tuning only when a bottleneck like latency, model size, or gradable outputs can't be solved without touching weights.

How do I decide whether to fine-tune an LLM or not?

Work down the stack: try prompt/context engineering first, then RAG if knowledge is proprietary or fresh, then agent skills if procedural know-how is missing. Only fine-tune if a specific, measurable bottleneck remains — real-time latency, need for a smaller/cheaper model, or programmatically gradable outputs. If you can't articulate the exact bottleneck, you don't need fine-tuning yet.

How do I stop a new frontier model from making my fine-tuned model obsolete?

Build a domain-specific benchmark before and after customization, and re-run it against every new frontier release — not just the model that existed at training time. This is the 'Moving Target Problem': a legal AI preferred 97% of the time in 2023 was surpassed by seven general-purpose models by 2025. Prefer non-weight-touching methods, which sidestep obsolescence entirely.

How does the Customization Stack compare to just fine-tuning everything?

Fine-tuning first is usually the wrong instinct. It carries hidden costs — labeled data collection, evaluation cycles, regression testing, and ongoing maintenance as frontier models evolve — and risks being leapfrogged before shipping. The Customization Stack solves most use cases with prompt engineering, RAG, or agent skills at a fraction of the cost, reserving fine-tuning for genuine, measurable bottlenecks.

When should I use RAG instead of expanding my context window?

Use RAG when your knowledge base is too large to fit in a single prompt or when you need to retrieve only the most relevant slices at query time. If a model can read all 500 relevant pages directly in its context window, you may not need RAG at all — 'Don't Bake What You Can Pass.' RAG scales retrieval when the corpus exceeds what fits in context.

What are agent skills and when should I use them?

Agent skills are folders of files (often markdown) that package procedural knowledge — how to do a task and which tools to use — that any general model loads on demand. Use them when the gap is procedural know-how, not missing facts. Example: instead of fine-tuning a model to write SQL against your schema, give it an SQL agent skill describing the schema and steps.

What is the difference between LoRA, distillation, and RFT?

LoRA trains a small adapter on top of a locked base model and is the default production fine-tuning method. Distillation trains a smaller 'student' model on outputs from a large 'teacher,' producing a cheaper, faster model. RFT (Reinforcement Fine-Tuning) uses a grader to score sampled answers and only works when outputs have a definitive right answer, like math.

When should I use RFT for fine-tuning?

Use RFT only when your outputs are programmatically gradable — there's a definitive right answer a grader can score, like math solutions or code that passes tests. RFT samples candidate answers, scores them with the grader, and updates the model to make high-scoring answers more likely. If outputs are subjective or open-ended with no clear correct answer, RFT won't work.

What results can I expect from applying this framework?

You'll typically solve most specialization needs without any training run, saving weeks of labeled-data collection and evaluation cycles. When fine-tuning is genuinely needed, you'll have a precise bottleneck and a benchmark to justify it. The biggest payoff is avoiding wasted investment in custom models that frontier releases render obsolete overnight.

Is fine-tuning still needed in 2025?

Rarely as a first choice, but sometimes yes. Expanded context windows, RAG, agent skills, and reasoning-at-inference-time now solve many cases that once required fine-tuning. Fine-tuning remains valid for real-time latency (via distillation to a small model), cost-driven model shrinking, or programmatically gradable outputs (via RFT) — but only after the non-weight-touching stack proves insufficient.

How do reasoning models change the fine-tuning decision?

Reasoning models do extended thinking at inference time, working through problems step by step before answering. This means specialization can come from how hard the model thinks at query time, not just from training months earlier. Before fine-tuning for accuracy, test whether a reasoning model with good context engineering already closes the gap.

// GET THIS SKILL — FREE