Frequently Asked Questions About IBM LLM Customization Stack Framework
21 answers covering everything from basics to advanced usage.
// Basics
What does 'non-weight-touching' mean in LLM customization?
Non-weight-touching methods specialize a model without changing its underlying parameters. Prompt/context engineering, RAG, and agent skills all leave the base model's weights untouched — they change what the model sees at query time, not what it learned in training. Fine-tuning is the only weight-touching method in the stack, and it sits at the bottom as a last resort.
What is context engineering and how is it different from prompt engineering?
Context engineering is the broader practice of assembling a structured prompt bundle — system prompt, relevant data, format guidelines, and task instructions — all packaged together. Prompt engineering often refers narrowly to wording the instruction well. Context engineering treats the entire input as an engineered artifact and is the mandatory first layer of the Customization Stack.
What is the Moving Target Problem in fine-tuning?
The Moving Target Problem is the risk that by the time your fine-tuned model ships, a new frontier release has already surpassed it, making the training investment obsolete. Frontier models improve rapidly, so a custom model's shelf life may be short. Always weigh a custom model's lifespan against the pace of frontier improvement before committing to a training run.
// How To
How do I apply context engineering as the first layer?
Start with a base model and assemble a complete prompt bundle: a clear system prompt defining the role, the relevant data the model needs, explicit format guidelines, and precise task instructions. Test whether this alone solves the use case before moving down the stack. Many domain tasks are fully solved here — never skip to weight-touching methods until this is verified insufficient.
How do I build an agent skill for a repeatable procedure?
Package the procedural knowledge into a folder of files — typically markdown — describing how to do the task and which tools to use. For an SQL skill, include the database schema, coding conventions, and a step-by-step generation procedure. Any general-purpose model loads the skill on demand when it sees a matching task, with no training required.
How do I set up RAG for proprietary knowledge?
Index your internal documents into a retrieval system. At query time, retrieve the most contextually relevant documents and pass them into the prompt alongside the user's question. This keeps knowledge fresh — update the index instead of retraining. Use RAG when knowledge is proprietary, internal, or updated frequently, since it solves the stale-weights problem without a training run.
How do I build a benchmark to justify fine-tuning?
Create a domain-specific evaluation suite that measures task performance on your actual use case, not a generic benchmark. Score your candidate approaches on it before and after customization, and re-run it against current frontier models — not just the frontier model at training time. Without this, you can't prove the fine-tuning investment actually beats a general model.
// Troubleshooting
My model doesn't know my domain — is that a fine-tuning problem?
Usually not. 'The model doesn't know our domain' is almost always a RAG or context engineering problem, not a fine-tuning one. If the knowledge is proprietary or fresh, add RAG. If it's missing procedural know-how, add an agent skill. Fine-tuning to inject facts is wasteful when you can retrieve or pass those facts at query time.
My RAG retrieval quality is poor — should I fine-tune instead?
Not yet. First optimize retrieval and context engineering — chunking strategy, embedding quality, reranking, and how retrieved content is formatted in the prompt. Only revisit fine-tuning if retrieval is provably insufficient after these optimizations. Fine-tuning won't fix a retrieval pipeline problem; it just adds cost and the Moving Target risk on top of an unsolved retrieval issue.
My fine-tuned model got worse at general tasks — what happened?
That's likely regression from fine-tuning, a hidden cost of touching weights. Focused training on a narrow dataset can degrade broader capabilities. This is one reason LoRA is preferred — it locks most original weights and trains only a small adapter. Always run regression tests against a broad benchmark, not just your domain task, to catch this before deployment.
My voice agent is too slow with a reasoning model — what do I do?
Real-time latency is a valid fine-tuning trigger. Use distillation: generate high-quality responses (including reasoning traces) from a large frontier teacher model, then fine-tune a small, fast student model on those outputs. Deploy the small model for latency-sensitive voice. Re-evaluate when smaller frontier models improve, since they may eventually match your distilled model.
// Comparisons
How does the Customization Stack compare to just using a bigger context window?
They're complementary. An expanded context window follows the 'Don't Bake What You Can Pass' principle — if a model can read 500 pages directly, don't train them into weights. But when the corpus exceeds context limits, RAG retrieves only relevant slices. The stack tells you when passing context is enough and when you need retrieval on top.
How does RAG compare to agent skills?
RAG solves missing knowledge — facts, documents, or fresh data the model lacks. Agent skills solve missing procedure — how to do a task and which tools to use. If the model has facts but doesn't know your workflow, use a skill. If it knows the workflow but lacks the data, use RAG. Many production systems combine both.
How does LoRA compare to full fine-tuning?
LoRA trains a small adapter on top of the base model with most original weights locked, making it cheaper, faster, and less prone to regression than full fine-tuning, which updates all parameters. LoRA is the dominant production fine-tuning method today. Full fine-tuning is rarely justified given LoRA's efficiency and lower risk of degrading general capabilities.
How does this framework compare to a generic 'just fine-tune it' approach?
A generic approach treats fine-tuning as the default, incurring labeled-data costs, evaluation cycles, and obsolescence risk. This framework inverts that: fine-tuning is a last resort behind three non-weight-touching layers. It forces you to articulate a precise bottleneck and build a benchmark before spending on training — avoiding the common trap of custom models leapfrogged by frontier releases.
How does distillation compare to RFT?
Distillation aims to make a smaller, cheaper, faster model by training a student on a large teacher's outputs — the goal is efficiency, not necessarily accuracy gains. RFT aims to improve accuracy on tasks with definitive right answers by scoring sampled outputs with a grader. Use distillation for latency/cost bottlenecks; use RFT only when outputs are programmatically gradable.
// Advanced
Can I combine multiple layers of the Customization Stack at once?
Yes, and production systems often do. A single deployment might use context engineering for role and format, RAG for proprietary knowledge, and an agent skill for a repeatable procedure — all on a base model with no fine-tuning. The stack is an ordered exhaustion sequence, not mutually exclusive layers. Combine freely, but still exhaust non-weight-touching layers before fine-tuning.
What are the hidden costs of fine-tuning I should budget for?
Beyond the training run, fine-tuning requires collecting labeled examples, evaluating results, regression testing to avoid degrading general capabilities, and ongoing maintenance as frontier models evolve. Each new frontier release forces a re-benchmark and possibly a re-train. These costs are recurring, not one-time, and must be explicitly justified against the specific bottleneck fine-tuning solves.
When is a smaller distilled model worth the fine-tuning investment?
When cost or latency at scale is the bottleneck and a large model is overkill. Distillation lets a small student inherit a teacher's capability at lower inference cost and speed. It's worth it for high-volume or real-time applications where per-query cost or response time dominates. Re-evaluate regularly, since smaller frontier models may eventually match your distilled model for free.
How do reasoning models affect whether I need fine-tuning for accuracy?
Reasoning models do extended step-by-step thinking at inference time, so specialization can come from how hard the model thinks at the question, not from training. Before fine-tuning to improve accuracy, test whether a reasoning model with strong context engineering already closes the gap. Inference-time reasoning often eliminates the need for a training run entirely.
How often should I re-run my benchmark against frontier models?
Every time a major new frontier model is released, and on a regular cadence otherwise. The legal AI case study showed a model preferred 97% of the time in 2023 was surpassed by seven general-purpose models by 2025. Benchmarking only against the frontier model at training time gives a false sense of durable advantage.