Frequently Asked Questions About Aishwarya Srinivasan LLM Fine-Tuning Decision Framework
22 answers covering everything from basics to advanced usage.
// Basics
What is fine-tuning in the context of LLMs?
Fine-tuning is post-training — adapting a pre-trained base model toward a specific task, domain, tone, or behavior. Unlike pre-training, which foundational labs perform on trillions of tokens, fine-tuning steers a model that already carries enormous knowledge. It changes how a model behaves and responds, not the base knowledge it was built on.
What is the difference between open-weight and closed-weight models?
Open-weight models (Llama, Mistral, Qwen, DeepSeek, GLM) expose their weights, so you can download, fine-tune, and deploy on your own infrastructure. Closed-weight models (GPT, Claude, Gemini) don't — you must use the provider's native fine-tuning service, submit data, and receive a fine-tuned API endpoint with no visibility into the process.
What is PEFT and why does it matter?
Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques that update only a small fraction of a model's parameters instead of all of them, drastically cutting compute, memory, and cost. LoRA and QLoRA are the main examples. PEFT matters because LLMs have billions to trillions of parameters, making full updates impractical for most teams.
What is the model-as-a-service reality in LLM fine-tuning?
It's the paradigm shift where the pre-trained model is created by a foundational lab and served to you via API, so you adapt it rather than train from zero. With LLMs the game moved from owning the entire training pipeline to steering existing knowledge toward your use case. This shapes every fine-tuning decision.
// How To
How do I establish a baseline before fine-tuning?
Run your task with optimized prompts and context injection (RAG) first, then record your eval metric. That number is your benchmark. It tells you whether fine-tuning is even necessary and gives you a comparable score to beat afterward. Skipping this step means you'll never know how much improvement fine-tuning actually added.
How do I set up an eval set for fine-tuning?
Choose a held-out set representative of your production task and a concrete metric — accuracy, win-rate, BLEU, safety pass-rate, or tone consistency. Lock this down before touching a single weight. Without a proper eval set and clear metric, fine-tuning is directionless and you won't know if it improved anything.
How do I apply DPO for preference alignment?
Create chosen/rejected response pairs from human rater comparisons, then run DPO directly on the language model in a single step — no separate reward model. DPO is the default for aligning open-source models to style, safety, and tone. Only escalate to RLHF if you have very large preference datasets and dedicated ML engineering capacity.
How do I fine-tune a reasoning model for math or code?
Use Reinforcement Fine-Tuning with Verifiable Rewards. The model generates multiple attempts per problem; correct answers are automatically scored and rewarded, wrong ones penalized — no human labelers needed. This works only when correctness can be checked programmatically, like running test cases on code. It's the technique behind OpenAI's o-series and DeepSeek R1.
// Troubleshooting
My fine-tuned model didn't improve over the baseline. What went wrong?
Diagnose in order: data quality first, then method choice, then hyperparameters. Do not add compute before fixing data — a bad dataset makes a good model worse. Audit for consistency, correctness, edge-case coverage, and format alignment with your target behavior. If data is clean, confirm you picked the right method for your task type.
Why can't I fine-tune GPT or Claude on my own GPUs?
Because closed-weight models don't expose their weights — you literally don't have the parameters to update. Attempting to fine-tune them on your own infrastructure is impossible. You must use the provider's native fine-tuning service, submit your data, and receive a fine-tuned API endpoint. The provider controls the method and gives no under-the-hood visibility.
My fine-tuning made the model worse. Why?
Almost always because of a bad dataset — data quality is the ceiling on every outcome. Fine-tuning with a bad dataset on a good model makes the model bad, not better. No technique compensates for poor data. Review your training data for inconsistency, incorrect labels, missing edge cases, and format mismatches before blaming the method.
I tried Reinforcement Fine-Tuning but it isn't working. Why?
This method only works when correctness can be verified automatically without human labelers — math problems, code with runnable tests. If your task lacks a programmatic ground-truth check, verifiable rewards can't be computed and the method fails. For subjective goals like tone or empathy, use DPO for preference alignment instead.
// Comparisons
QLoRA vs Full Fine-Tuning — which should I use?
Start with QLoRA. It quantizes the frozen base to 4-bit and trains small LoRA matrices, letting you fine-tune a 70B model on a single H100. Full Fine-Tuning updates every parameter and offers maximum flexibility but is expensive, slow, and demands serious GPU infrastructure. Escalate to Full Fine-Tuning only when QLoRA misses your quality bar — rare in practice.
DPO vs RLHF — what's the practical difference?
DPO skips training a separate reward model and optimizes the language model directly on preference pairs in one step — simpler, cheaper, and the default for open-source models. RLHF trains a reward model on human comparisons then runs PPO to guide the language model — powerful but complex and expensive. Use RLHF only with large, high-quality preference data and dedicated ML engineering capacity.
Fine-tuning vs RAG — when do I use each?
Use RAG to inject new factual knowledge and fine-tuning to adapt behavior, style, tone, or task performance. Fine-tuning doesn't reliably teach new facts; RAG changes what information the model can access at inference time. For a company knowledge base, RAG usually wins. For consistent formatting or domain vocabulary, fine-tuning wins. They can also be combined.
How does this framework compare to just calling a fine-tuning API blindly?
A blind API call skips baseline measurement, evaluation setup, and method selection — so you can't tell if fine-tuning helped, may waste compute on the wrong method, and risk making the model worse with bad data. This framework forces a prompt/RAG benchmark, an eval metric, model-type awareness, and task-based routing, giving you genuine engineering depth instead of guesswork.
// Advanced
Does fine-tuning teach a model new factual knowledge?
No — fine-tuning adapts behavior, style, and task performance, not factual recall. Assuming it teaches new facts is a common and costly mistake. For knowledge injection, retrieval-augmented generation (RAG) is typically the right tool. Reserve fine-tuning for tone, formatting, domain adaptation, preference alignment, and reasoning improvements.
How do I decide between domain adaptation and preference optimization?
Ask what 'better' means for your task. If you need the model to perform a specific task or adopt domain vocabulary and formatting, that's domain/task adaptation — use QLoRA. If you need it to align to human judgments about style, safety, or tone, that's preference optimization — use DPO. If success is a deterministic correct/incorrect answer, use verifiable rewards.
Why is parameter scale central to method selection?
Traditional ML models had millions of parameters; LLMs range from a few billion to over a trillion. Updating even a fraction of a trillion-parameter model isn't something you can casually do on a GPU. This scale is exactly why PEFT techniques like QLoRA exist and why method selection — not raw compute — determines whether fine-tuning is feasible.
How does LoRA actually work under the hood?
LoRA freezes the original model weights and injects small trainable low-rank matrices into specific layers. Only those matrices are trained during fine-tuning. At inference time they combine with the frozen weights to produce adapted behavior. QLoRA adds 4-bit quantization of the frozen base, cutting memory further so you can fine-tune very large models on a single GPU.
What infrastructure do I need for each fine-tuning method?
QLoRA can run a 70B model on a single H100 thanks to 4-bit quantization. Full Fine-Tuning requires serious multi-GPU infrastructure. DPO needs moderate resources plus preference pair data. RLHF demands the most — reward model training plus a PPO loop and engineering capacity. Closed-weight fine-tuning needs no GPUs of your own; the provider runs it.
Can I combine multiple fine-tuning methods?
Yes — production pipelines often stack them. A common pattern is domain adaptation with QLoRA followed by preference alignment with DPO for tone and safety. Frontier reasoning models combine supervised fine-tuning with reinforcement fine-tuning on verifiable rewards. Always define eval metrics for each stage and audit data quality before every job so you can attribute gains correctly.