How to Choose LoRA, Distillation, or RFT for Fine-Tuning
For ML engineers and applied scientists · Based on IBM LLM Customization Stack Framework
// TL;DR
ML engineers are often handed 'make it a specialist' tasks and reach straight for fine-tuning. The IBM LLM Customization Stack Framework gives you a technical discipline: verify prompt/context engineering, RAG, and agent skills are insufficient before touching weights, then select the right fine-tuning method for the bottleneck — LoRA for general adaptation, distillation for smaller/faster models, RFT for programmatically gradable outputs. It also enforces benchmarking against current frontier models, so your training runs stay defensible instead of obsolete.
When should an ML engineer actually reach for fine-tuning?
Only when a concrete, measurable bottleneck survives the non-weight-touching stack. Before writing a training script, confirm that context engineering, RAG, and agent skills genuinely can't close the gap. If your model lacks facts, that's RAG. If it lacks procedure — like how to write SQL against a specific schema — that's an agent skill. Fine-tuning is justified only for bottlenecks like real-time latency, the need for a smaller/cheaper distilled model, or outputs you can programmatically grade.
How do you choose between LoRA, distillation, and RFT?
Match the method to the bottleneck:
- LoRA (Low-Rank Adaptation) — Your default. Train a small adapter on top of the base model with most original weights locked. It's the dominant production method: cheap, fast, and less prone to catastrophic regression than full fine-tuning.
- Distillation — When you need a smaller, cheaper, or faster model. Generate high-quality outputs (including reasoning traces) from a large teacher model, then fine-tune a smaller student on those outputs. Ideal for the sub-500ms voice-agent case where frontier reasoning models are too slow.
- RFT (Reinforcement Fine-Tuning) — Only when outputs are programmatically gradable. Build a prompt dataset plus a grader; the model samples candidate answers, the grader scores them, and training makes high-scoring answers more likely. Perfect for math tutoring or code that passes tests. If there's no definitive right answer, RFT won't work.
How do you avoid shipping an already-obsolete model?
Build a domain-specific benchmark that measures performance on your actual task. Test your fine-tuned model against current frontier models, not just the frontier baseline at training time. The Moving Target Problem is real: a legal AI preferred 97% of the time in 2023 was surpassed by seven general models by 2025. Re-run your benchmark on every major frontier release and be willing to retire a custom model when a base model catches up.
What technical mistakes should you watch for?
Common failure modes include: using RFT on subjective tasks with no gradable answer; treating a retrieval-quality problem as a fine-tuning problem (fix chunking, embeddings, and reranking first); and ignoring regression — narrow fine-tuning can degrade general capabilities, which is why LoRA's locked weights help. Also remember 'Don't Bake What You Can Pass': if a document fits in an expanded context window, there's no reason to train it into the weights.
How do reasoning models change your engineering plan?
Reasoning models do extended thinking at inference time, deriving specialization from how hard they think at the moment of the question. Before you commit to fine-tuning for accuracy, benchmark a reasoning model with strong context engineering — it may already meet your target and save the training run entirely. Fine-tuning for accuracy is often the least necessary reason to touch weights in the reasoning-model era.
Next step: For your current specialization task, write a one-line bottleneck statement and map it to LoRA, distillation, or RFT — or back up the stack to RAG or an agent skill. Then draft the benchmark you'll use to validate the model against today's frontier releases before you spend a single GPU-hour.
// FREQUENTLY ASKED QUESTIONS
When is RFT the wrong choice for fine-tuning?
RFT is wrong whenever outputs can't be programmatically graded — there's no definitive right answer a grader can score. Subjective tasks like tone, style, or open-ended writing don't fit. RFT relies on sampling candidate answers and scoring them; without a reliable grader, training has no signal to optimize toward. Use it for math, code with tests, or other verifiable outputs.
Why is LoRA the default instead of full fine-tuning?
LoRA trains a small adapter while locking most original weights, making it cheaper, faster, and far less prone to regression than full fine-tuning, which updates every parameter. It's the dominant production method today because it delivers domain adaptation without the cost and general-capability degradation risk of retraining the whole model.
Should I fine-tune to improve accuracy on my domain?
Usually test alternatives first. Reasoning models with extended inference-time thinking, combined with strong context engineering and RAG, often close accuracy gaps without any training. Fine-tuning for accuracy is frequently unnecessary and carries the Moving Target risk. Reserve weight-touching for latency, model-size, or gradable-output bottlenecks that the stack genuinely cannot solve.