Frequently Asked Questions About Zen van Riel Local AI Fine-Tuning Pipeline
21 answers covering everything from basics to advanced usage.
// Basics
What is the prompt ceiling in fine-tuning?
The prompt ceiling is the performance limit of pure prompt engineering — the point where further prompt iteration cannot overcome the base model's trained behaviours. When your output still sounds robotic or off-style after iterating on prompts, you've hit it. At that point, fine-tuning is the only viable path to bake the desired behaviour directly into the model's weights.
What does 'garbage in, garbage out, amplified' mean for fine-tuning?
A language model cannot self-correct dirty training data. Spelling errors, wrong formats, and mismatched prompt-response pairs get baked permanently into the fine-tuned model — and the effect is amplified because the model learns from them as ground truth. This is why dataset engineering is not a shortcut step but the single step most responsible for final model quality.
What is chat-format correspondence and why does it matter?
Chat-format correspondence means your training data must mirror the conversational structure the model uses at inference time. Raw documents like transcripts and docs aren't usable as-is — they must become prompt-response pairs matching how you'll actually query the model. Skipping this transformation produces a broken fine-tuned model that doesn't respond correctly to real prompts.
What is the decision ladder before fine-tuning?
The decision ladder is a sequence of cheaper options to exhaust before fine-tuning: (1) write a better prompt, (2) add RAG, (3) build an agentic loop. Only proceed to fine-tuning if all three fail to meet your performance bar, or if owning an in-house model is a strategic advantage. Document why each tier failed — that evidence guides your later evaluation criteria.
// How To
How do I run the decision ladder in practice?
Start by asking whether a better system prompt solves the problem — if yes, stop. If not, add RAG and re-test. If RAG is insufficient, try an agentic loop. Only if all three fail, or if an in-house model is itself strategic, proceed to fine-tuning. At each step, document the specific failure so you know exactly what your fine-tuned model must fix.
How do I generate prompt-response pairs from raw documents?
Use a local language model to auto-generate plausible questions for each text snippet, then pair that snippet as the answer. Tailor the pair structure to your end use case — if the goal is blog writing, every prompt should follow a consistent template like 'Write a blog post on this topic: {topic}' so the formatting rules become embedded in the fine-tuned model.
How do I validate LoRA parameters before a full training run?
Run a short test training job first. Since a full run takes 2–3 hours on a high-end consumer GPU, discovering a wrong parameter setting only after committing wastes multiple hours. Validate both your dataset and your LoRA configuration on a small subset, confirm the model behaves as expected, then scale to the full training job.
How do I disable chain-of-thought thinking in a fine-tuned model?
Explicitly disable behaviours you don't want, such as extended chain-of-thought 'thinking' mode, during the LoRA training phase. If your target model ships with reasoning modes you don't need, configure them off in your training setup so the fine-tuned output goes straight to the answer without unwanted reasoning traces.
What base model should I choose for fine-tuning?
Choose an open-source base model whose size matches your hardware — size determines VRAM requirements and training time. The pipeline references models like Qwen 3.5 27B, which needs 14+ GB VRAM. Balance capability against what fits in your dedicated VRAM without offloading, and factor in the 2–3 hour training runs per iteration on consumer GPUs.
// Troubleshooting
Why is my fine-tuning grinding to a halt during training?
You're almost certainly relying on RAM offloading because the model doesn't fit in dedicated VRAM. RAM offloading grinds fine-tuning to a halt and often fails entirely — dedicated GPU VRAM is a hard requirement. Choose a smaller base model, use a GPU with more VRAM, or pair GPUs so the model and adapter fit fully in VRAM.
My fine-tuned model baked in spelling errors — what happened?
You fed raw, uncleaned data — likely auto-generated transcripts or OCR output — directly into training. The model cannot self-correct bad input and bakes those errors in permanently. There's no post-training fix; you must clean the dataset (spelling, punctuation, transcription artefacts) and re-run the full fine-tuning pipeline from the dataset engineering step.
The fine-tuned model doesn't follow my house style — how do I diagnose it?
Run a structured evaluation: send identical test prompts to both the vanilla base model and your fine-tuned model, then compare on the criteria that motivated fine-tuning. If the fine-tuned model doesn't show unprompted adherence to your style, the failure usually traces back to dataset engineering — inconsistent prompt templates or insufficient on-brand examples — or to LoRA misconfiguration.
My model fits in VRAM but trains extremely slowly — why?
Model fit and fine-tuning throughput are separate concerns. A model fitting in VRAM does not guarantee acceptable training speed. Throughput depends on the GPU's compute capability, memory bandwidth, and CUDA/ROCm support. Always benchmark a short run before committing hardware to a full multi-hour training job — don't assume fit equals speed.
// Comparisons
How does fine-tuning compare to just writing a better prompt?
A better prompt can guide a model but cannot escape behaviours baked in during its original training. Prompts are free and instant; fine-tuning costs at minimum a weekend plus significant hardware. Use prompts first — the decision ladder demands it. Fine-tune only when prompt iteration has hit its ceiling and the output still sounds generic or off-persona.
Should I use fine-tuning or an agentic loop?
An agentic loop lets the model decide whether to pull more information, invoke tools, or delegate to another agent — it's the final option to try before fine-tuning. Choose the agentic loop when the problem is about accessing or orchestrating dynamic information. Choose fine-tuning when the problem is embedded behaviour — consistent tone, style, or stable domain knowledge — that no orchestration can fix.
Is Nvidia really necessary, or can I use AMD?
Nvidia + CUDA is the first-class citizen for fine-tuning because Python libraries and frameworks interact with it efficiently. AMD + ROCm is a viable fallback for hobbyists but delivers variable results and compatibility. If you're serious about speed and reliability, choose Nvidia; if you're experimenting on a budget with AMD hardware, ROCm can work with tempered expectations.
// Advanced
When should I use a hybrid fine-tuning plus RAG approach?
Use a hybrid approach when you have both stable and dynamic knowledge. Fine-tune the stable corpus — laws unlikely to change, house style, persona voice — so it's baked in for reliability and speed. Then layer RAG on top for recent updates like statute amendments or new case law. This avoids retraining every time your dynamic knowledge changes.
How many tokens do I need for a larger base model?
The pipeline gives 1–2 million+ raw tokens as the target for an 8B model before transformation. Larger models generally benefit from more high-quality data, but with LoRA targeting only 0.5–1.5% of parameters, data requirements stay dramatically lower than full fine-tuning. Prioritize data cleanliness and chat-format correctness over sheer volume — quality drives final model quality.
How do I build an evaluation suite for a fine-tuned model?
Write a suite of test prompts representative of your real use case, then run identical prompts on both the vanilla base model and your fine-tuned model. Score both on the specific criteria that motivated fine-tuning — tone, brevity, accuracy, format. Use the results to diagnose failures back to dataset engineering or LoRA parameters, then iterate on those, not on post-hoc prompting.
Why should I iterate on data instead of prompting the fine-tuned model?
Wrong output almost always traces back to dataset engineering errors or LoRA parameter misconfiguration — the root cause is in the model, not the prompt. Patching it with post-hoc prompting reintroduces the prompt ceiling you fine-tuned to escape. Fix the actual cause by iterating on your dataset or training parameters, then re-run and re-evaluate.
Can I pair multiple GPUs to fit a larger model in VRAM?
Yes — pairing GPUs is one way to reach the VRAM capacity a larger model needs, since VRAM is the hard constraint. Confirm your fine-tuning framework supports multi-GPU setups and that the model plus LoRA adapter fits fully in the combined dedicated VRAM. The goal is always to avoid RAM offloading, which cripples training throughput.