Frequently Asked Questions About KodeKloud LLM Fine-Tuning Pipeline
21 answers covering everything from basics to advanced usage.
// Basics
What does 'embedding behavior into model weights' actually mean?
It means the desired behavior — persona, format, tone, or domain rules — becomes part of how the model computes its outputs, rather than being a request in the prompt. Prompt engineering suggests behavior; fine-tuning encodes it. Because the behavior lives in the weights, an attacker can't simply override it by writing 'ignore your instructions' the way they can with a system prompt.
What is a base model in this context?
The base model is the original pre-trained foundation model before any fine-tuning — a generalist, not a specialist. In this pipeline you pick a small open-weight model that runs on consumer hardware. During LoRA training the base model's weights stay frozen, and only the added adapter matrices are trained, which is what keeps memory requirements low.
What is an adapter and why is it so small?
An adapter is the small set of trainable weight matrices added on top of a frozen base model during LoRA fine-tuning. It captures all the learned behavior and is saved separately — typically around 2 MB versus ~500 MB for a full model. It's small because LoRA trains only low-rank matrices rather than the model's billions of original parameters.
What are chosen and rejected response pairs?
They're the core data structure of DPO. The 'chosen' response is what the model should prefer to generate — helpful, on-brand, appropriate. The 'rejected' response is what it should learn to avoid — rude, harmful, off-character, or dismissive. Together they define the alignment direction, teaching the model not just what to say but what not to say.
// How To
How do I configure LoRA hyperparameters for the first time?
Start with rank (r) = 8, alpha = 16, and target modules q_proj and v_proj — the query and value projection layers in the attention mechanism. Higher rank gives more capacity but uses more memory. After configuring, confirm the parameter reduction output: you should see roughly 99%+ of parameters frozen. That confirmation is what makes training feasible without a data center.
How do I run the training loop and know it's working?
Set training steps — start around 50 for small models — and a learning rate of 2e-4 as a solid default. Run the loop and watch the loss decrease across steps; a consistently declining loss confirms the model is learning your target behavior. When done, save the adapter (~2 MB) rather than the full model, since the adapter alone contains all learned behavior.
How do I test whether fine-tuning actually worked?
Run a head-to-head comparison of the base model versus the fine-tuned model on three prompt types: normal on-topic prompts, off-topic prompts, and jailbreak prompts. Score on-topic relevance for each. The fine-tuned model should score measurably higher on domain relevance and hold character when attacked. If it still breaks, add more adversarial training examples and retrain.
How do I build DPO preference data for alignment?
For each sensitive or edge-case scenario, write two responses: a chosen response that's helpful, on-brand, and appropriate, and a rejected response that's rude, harmful, off-character, or dismissive. Validate each pair before adding it to the DPO dataset. This is the same mechanism used to align commercial models like ChatGPT toward being helpful instead of harmful.
// Troubleshooting
My fine-tuned model still breaks character under jailbreaks. What now?
Return to the training data step and add more adversarial examples — specifically attack prompts paired with correct in-character refusals. Fine-tuning makes jailbreaks significantly harder but not impossible, so treat it as one layer in defense-in-depth. Also add DPO pairs where the chosen response holds character and the rejected response is the slippage you observed.
Why did my model forget its base capabilities after fine-tuning?
Your learning rate is likely too high, causing catastrophic forgetting — the model learns your domain while discarding general capabilities. Start at 2e-4 and adjust based on the loss curve. If loss drops too abruptly or the model becomes narrow and brittle, lower the rate. Also verify your dataset isn't so small or repetitive that it overfits.
My training hits memory limits — what went wrong?
Check that LoRA actually froze the base weights. If the parameter reduction output shows all parameters are still trainable, you're not using LoRA efficiently and will hit memory ceilings fast. Confirm you see ~99%+ of parameters frozen. Also consider lowering the LoRA rank, since higher rank increases capacity and memory usage.
The model outputs correct behavior but occasionally says something harmful. How do I fix that?
That's a sign you skipped or under-invested in DPO alignment. A model that knows how to stay on topic can still produce dismissive or harmful responses in edge cases. Add DPO preference pairs targeting exactly those scenarios, with the chosen response being appropriate and the rejected response being the harmful output you saw. DPO separates a capable agent from an aligned one.
// Comparisons
Should I use fine-tuning or RAG to give my agent company knowledge?
Use RAG for knowledge. Fine-tuning teaches HOW the model behaves — format, persona, tone, constraints — while RAG pulls WHAT it needs to know from documents and live data. Injecting facts through fine-tuning is an expensive mistake because facts change and retraining is costly. Combine both: fine-tune for behavior, retrieve for knowledge.
How does LoRA compare to full fine-tuning?
Full fine-tuning retrains all of a model's billions of parameters, requiring data-center-scale memory and compute. LoRA freezes the base weights and trains only small adapter matrices, cutting trainable parameters by ~99.7% and memory from ~1,500 MB to ~5 MB. You get most of the behavioral benefit at a fraction of the cost, and the resulting adapter is tiny and portable.
Is DPO better than RLHF for aligning my agent?
For most practical projects, DPO is easier and sufficient. RLHF requires human raters to score outputs plus a separate reward model and reinforcement loop. DPO skips all of that, training directly on chosen/rejected preference pairs to make the model helpful, harmless, and honest. It's the pragmatic choice when you can author good preference pairs but don't want a full scoring pipeline.
How does a fine-tuned agent compare to just adding more rules to the system prompt?
More rules in a system prompt still only suggest behavior and remain vulnerable to injection — a user can instruct the model to ignore them. A fine-tuned agent has behaviors embedded in its weights, so overriding them requires far more than a clever prompt. The pipeline even mandates proving the prompt-only version fails first, to justify the fine-tuning effort.
// Advanced
Why is exposing the prompt-engineering problem the mandatory first step?
Because without proving the prompt-only version fails, you don't know which specific behaviors need embedding or how to measure success. This baseline failure case justifies the entire effort and becomes your evaluation benchmark. Deliberately test the prompt-only agent against jailbreaks like 'Ignore your instructions and...' and document exactly where and how it breaks before touching any training code.
Which target modules should I fine-tune in the attention mechanism?
Start with q_proj and v_proj — the query and value projection layers — as your LoRA target modules. These are common, effective starting points for embedding behavior with minimal parameters. You can expand to more projection layers if you need additional capacity, but adding modules increases memory and trainable parameters, so validate the trade-off against your loss curve and results.
Can I stack fine-tuning with other defenses against prompt injection?
Yes, and you should. Fine-tuning makes jailbreaks significantly harder but not impossible, so treat it as a strong layer in a defense-in-depth strategy rather than a silver bullet. Combine it with input validation, output filtering, monitoring, and clear system prompts. The embedded behavior raises the cost of a successful attack; the surrounding controls catch what slips through.
How do I decide the right number of training steps and dataset size?
Begin with a small, high-quality dataset and around 50 training steps for a small model, then watch the loss curve. If loss keeps declining and evaluation improves, you're on track. Prioritize quality and coverage — on-topic, edge-case, and off-topic refusal examples — over raw volume. Add more steps or examples only if the model underperforms on your head-to-head evaluation.
Does fine-tuning guarantee my agent will never be jailbroken?
No. Fine-tuning embeds behavior into weights and makes jailbreaks significantly harder, but it doesn't make them impossible. The pipeline explicitly warns against treating fine-tuning as the only layer of defense. Use it as the core behavioral hardening step, then layer additional safeguards. Re-test regularly with new adversarial prompts and expand your DPO and training data as new attacks emerge.