Frequently Asked Questions About Tech With Tim LLM Fine-Tuning to Ollama Pipeline
20 answers covering everything from basics to advanced usage.
// Basics
What does LoRA do during fine-tuning?
LoRA (Low-Rank Adaptation) adds small trainable adapter layers on top of the frozen base model's weights instead of retraining the entire model. This makes fine-tuning fast and data-efficient, letting you train on modest hardware like a free Colab T4. You don't need to understand every LoRA parameter — Unsloth's defaults work well for most tasks.
What is the format_prompt function and why do I need it?
The format_prompt function merges each input/output training example into a single string, including a terminal end-of-text token so the model learns where each example ends. You write it yourself and customise it to match your dataset structure. Its output is wrapped in a Hugging Face Dataset with a 'text' field that SFTTrainer consumes.
What is 4-bit quantisation and should I use it?
4-bit quantisation (load_in_4bit=True) reduces a model's memory footprint by representing weights in 4-bit precision instead of 16 or 32-bit. This lets larger models fit in limited GPU RAM during fine-tuning. Enable it when working on a free Colab T4, especially with mid-size models, to avoid running out of GPU memory.
// How To
How many training examples do I actually need?
Aim for hundreds at minimum, ideally thousands. A small dataset of around 500 examples with a small model will produce inconsistent outputs. More examples always improve reliability, but training time scales with dataset size. Start with what you have to validate the pipeline, then expand the dataset to improve consistency.
How do I write a Modelfile for my fine-tuned model?
Create a plain-text file named exactly 'Modelfile' (capital M) in the folder with your .gguf file. Include FROM ./yourfile.gguf, PARAMETER entries (temperature, top_p, and a stop token matching your training end-of-text token), a TEMPLATE block defining the user/assistant turn structure, and a SYSTEM message setting the model's persona.
How do I register and test my model in Ollama?
Run 'ollama create <model-name> -f Modelfile' from the directory containing both the Modelfile and .gguf file. Verify with 'ollama list' — your named model should appear. Then run 'ollama run <model-name>' and paste a real input from your dataset to confirm end-to-end operation matches expected output.
How do I generate synthetic training data if I don't have enough real examples?
Use an LLM like GPT-4 or Claude to generate input/output pairs matching your task's format. Provide a few real examples as a template, then ask it to produce many variations. Review the synthetic examples for quality — data quality is the ceiling of your fine-tuned model, so garbage synthetic data still produces a garbage model.
// Troubleshooting
Why do my imports fail after installing Unsloth in Colab?
Because you skipped the mandatory runtime restart after the pip install cell. Colab warns that packages were previously imported and shows a 'Restart Session' button — you must click it before proceeding, or imports fail silently and are hard to debug. This is a required step, not optional.
Why is my training failing or producing malformed output?
The most common cause is forgetting to convert output JSON objects to strings with JSON.dumps before writing them to the dataset. Every 'output' value must be a string. Another cause is a mismatch between max_seq_length in the SFTTrainer and the value set when loading the base model — they must match exactly.
Why does Colab time out or run out of GPU memory during training?
You're likely using too large a base model for the free T4 GPU. Start with a small model like Phi-3 Mini to validate your pipeline and data, then scale up to a larger model only after confirming everything works. Also ensure load_in_4bit=True is set to reduce memory footprint.
Why does the loss not decrease during training?
Flat or non-decreasing loss usually means a data formatting problem — check that your format_prompt function is correctly merging inputs and outputs and that the Dataset text field is populated. Also verify your examples are meaningful and consistent. Monitor the loss window during training; steadily decreasing loss confirms the model is learning.
// Comparisons
How does fine-tuning with Unsloth compare to using OpenAI's fine-tuning API?
Unsloth fine-tunes open-source models for free on Colab and lets you run them locally via Ollama with full data privacy and no per-query cost. OpenAI's fine-tuning API is easier to use but ties you to their models, charges for training and inference, and sends your data to their servers. Choose Unsloth for privacy, cost control, and local deployment.
Is fine-tuning better than RAG (retrieval-augmented generation)?
They solve different problems. Fine-tuning teaches the model a consistent format, style, or narrow behaviour, restructuring its knowledge. RAG injects external documents into the prompt at query time for factual recall. Use fine-tuning for format and tone consistency; use RAG when facts change frequently or you need source-grounded answers. Many production systems combine both.
Should I pick a small or large base model?
Start with the smallest model that can plausibly learn your task, like Phi-3 Mini, for fast iteration and to validate your pipeline and data. Upgrade to a larger base model like Llama 3.1 8B only after confirming everything works. Smaller models train faster and cost less to run; larger models perform better on complex tasks.
// Advanced
Can I use a fine-tuned model as a general-purpose assistant?
No. Fine-tuning makes a model better at your specific task but measurably worse at general tasks — this is the specialisation trade-off and it's expected. Don't use a fine-tuned model as a drop-in replacement for a general assistant. Fine-tune only when you need domain depth, and keep a general model available for broad queries.
How do I match the stop token between training and Ollama?
Whatever end-of-text token you appended in your format_prompt function during training (e.g., <|endoftext|>) must be set as the stop PARAMETER in your Modelfile. If they don't match, the model hallucinates text beyond the intended response boundary. Keep a note of the exact token string and reuse it verbatim in the Modelfile.
How do I verify the model works before downloading the GGUF file?
Run inference inside Colab immediately after training. Construct a test message mirroring a real dataset example and check the output matches your expected format. Test several different inputs. Only proceed to GGUF export and download once behaviour is satisfactory — re-downloading after discovering problems wastes 20–30 minutes each time.
What max_seq_length should I set?
Set max_seq_length to the maximum token length your inputs will reach, for example 2048. Critically, the value you set when loading the base model must exactly match the value passed to the SFTTrainer, or training will error out. Choose a length that comfortably fits your longest formatted training example without wasting memory.
Can I fine-tune a model to output structured JSON reliably?
Yes — this is one of the best use cases. Create input/output pairs where outputs are JSON strings (converted via JSON.dumps), fine-tune a small model, and set the Modelfile stop token to match your training end-of-text token. The resulting model reliably returns structured JSON for new inputs, far more consistently than prompting a general model.
How do I iterate quickly when my fine-tuned model isn't good enough?
Keep your base model small during iteration for fast training cycles. Improve the dataset first — add more examples, fix inconsistent formatting, remove bad pairs — since data quality is the ceiling. Re-run training and verify inside Colab before exporting. Only scale to a larger base model once your data and pipeline consistently produce good results.