Tech With Tim LLM Fine-Tuning to Ollama Pipeline

Last updated: 11 July 2026

Fine-tune any open-source LLM on your own domain-specific data and deploy it locally via Ollama, without training from scratch.

// TL;DR

The Tech With Tim LLM Fine-Tuning to Ollama Pipeline is a step-by-step workflow for fine-tuning an open-source language model (like Phi-3 Mini or Llama 3.1) on your own domain-specific data using Unsloth and Google Colab's free T4 GPU, then deploying it locally with Ollama in GGUF format. Use it when you need a model that consistently outputs a specific format or style, works with data the base model has never seen, or when you want to replace a large expensive API model with a smaller, cheaper, specialized local one. No training from scratch required.

Framework

// When should you use the LLM fine-tuning to Ollama pipeline?

Use this skill when you need a model that consistently outputs a specific format or style, works with domain-specific data the base model has never seen, or when you want to replace a large expensive model with a smaller specialized one.

// What do you need before fine-tuning your own LLM?

Training Datasetrequired
A JSON file of input/output pairs (minimum hundreds, ideally thousands of examples) representing the exact task you want the model to learn. Each example must have an 'input' field and an 'output' field.
Base Model Choicerequired
The open-source model to fine-tune (e.g., Phi-3 Mini, Llama 3.1, Mistral, Mixtral). Smaller models train faster; larger models perform better. Must be available on Unsloth.
Task Descriptionrequired
A clear description of what you want the fine-tuned model to do — the specific domain, output format, or behaviour you are training for.
System Prompt
A short instruction string for the Modelfile (e.g., 'You are a helpful AI assistant') that sets the model's persona in Ollama.
Google Colab Access
A Google account to access Colab's free T4 GPU runtime. Required unless you own a GPU of 4080 / 4090 tier or better.

// What core principles make fine-tuning succeed?

Data Quality is the Ceiling

If you have bad data, you will have a poorly fine-tuned model. The training dataset is the single most important step. Take your time gathering correct, representative input/output pairs before touching any code.

Fine-Tuning vs. Parameter Tuning

Parameter tuning (adjusting temperature, top-K, etc.) is like adjusting your car's radio. Fine-tuning is like teaching your car to drive in a completely different neighborhood — it restructures the model's knowledge, not just its output style.

The Specialisation Trade-off

Fine-tuning makes models better at your specific task but worse at general tasks. This is expected and acceptable. Fine-tune only when you need domain depth, not general capability.

Start with a Small Base Model

Use the smallest model that can plausibly learn your task (e.g., Phi-3 Mini) for fast iteration. Upgrade to a larger base model only after confirming your pipeline and data work correctly.

GGUF is the Ollama Bridge

Ollama only understands the GGUF model format. Always export your fine-tuned model as GGUF (via Unsloth's save_pretrained_gguf) before attempting to load it locally.

// How do you fine-tune an LLM and deploy it to Ollama step by step?

1
Assemble your training dataset as a JSON file of input/output pairs
Every example must have an 'input' key and an 'output' key. Output values must be strings (use JSON.dumps if your output is a JSON object). Aim for hundreds to thousands of examples — more examples means better performance but longer training time. You can use an LLM to generate synthetic examples if real data is scarce.
2
Open the Unsloth fine-tuning notebook in Google Colab and connect to a T4 GPU runtime
Go to Runtime > Change Runtime Type and select T4 GPU + Python 3. Confirm via the GPU check cell that CUDA is available and the Tesla T4 is shown. Local training is viable only if you have a GPU of 4080 / 4090 tier or stronger; otherwise Colab is strongly recommended.
3
Upload your training dataset JSON file to Colab's file system
Use the Files panel on the left sidebar > Upload button. Note the exact filename — you will reference it in the data-loading cell.
4
Install Unsloth and all dependencies, then restart the Colab runtime
Run the pip install cell. After installation completes, Colab will warn that packages were previously imported — click 'Restart Session' before proceeding. This is mandatory or imports will fail.
5
Select and load your base model via Unsloth's FastLanguageModel
Set model_name to any Unsloth-supported model (Phi-3 Mini, Llama 3.1, Mistral, Mixtral, etc.). Set max_seq_length to the maximum token length your inputs will reach (e.g., 2048). Set load_in_4bit=True for 4-bit quantised models. Leave dtype=None for auto-detection.
6
Write a format_prompt function that merges each input/output pair into a single training string
The string must contain the input, a newline, the output, and a terminal end-of-text token (e.g., <|endoftext|>) so the model knows where the example ends. Apply this function across all examples and wrap the result in a Hugging Face Dataset object with a 'text' field.
7
Apply LoRA adapters to the loaded model using Unsloth's get_peft_model
LoRA (Low-Rank Adaptation) adds trainable layers on top of the frozen base model — this is what makes fine-tuning fast and data-efficient. You do not need to understand every parameter; defaults work. Use Colab's built-in Gemini 'Explain Code' feature if you need parameter clarification.
8
Initialise the SFTTrainer and run model training
Pass the model, tokenizer, dataset, and dataset_text_field='text' to the SFTTrainer. Ensure max_seq_length matches Step 5. Training time scales with dataset size and base model size. A small model on ~500 examples takes roughly 10 minutes on a T4. Monitor the loss window — decreasing loss confirms learning is happening.
9
Run inference inside Colab to verify the fine-tuned model behaves correctly before downloading
Construct a test message that mirrors a real example from your dataset. Check that the output matches the expected format. Test several different inputs. Only proceed to download if behaviour is satisfactory — re-downloading after discovering issues wastes significant time.
10
Export the fine-tuned model in GGUF format using model.save_pretrained_gguf
This saves a file named something like 'unsloth.Q4_K_M.gguf'. After saving to Colab's filesystem, run the download cell to pull it to your local machine. Both steps combined can take 20–30 minutes depending on model size and internet speed. Be patient.
11
Create a Modelfile that points Ollama to your downloaded GGUF file
In your terminal, navigate to the folder containing the .gguf file. Create a file named exactly 'Modelfile' (capital M). It must contain: FROM ./<your-filename>.gguf, PARAMETER entries (temperature, top_p, stop tokens matching your training format), a TEMPLATE block defining the user/assistant turn structure, and a SYSTEM message. The stop token must match whatever end-of-text token you used during training.
12
Register the model in Ollama using 'ollama create <model-name> -f Modelfile'
Run this command from the directory containing both the Modelfile and the .gguf file. Verify registration with 'ollama list' — your named model should appear. Then run 'ollama run <model-name>' and paste a real input to confirm end-to-end operation.

// What are real-world examples of this fine-tuning pipeline in action?

A developer wants to extract structured product data (name, price, category, manufacturer) from raw HTML snippets and needs consistent JSON output every time.

Generate 500+ input/output pairs where inputs are HTML fragments and outputs are JSON strings. Fine-tune Phi-3 Mini using the pipeline. The format_prompt function wraps each pair with an end-of-text token. After training, the Modelfile stop parameter is set to the same end-of-text token. The resulting Ollama model reliably returns structured JSON for any new HTML input.

A legal-tech startup needs a model that interprets internal contract clauses using company-specific terminology that no public LLM has been trained on.

Compile hundreds of clause/interpretation pairs from internal legal documents. Fine-tune a mid-size open-source model (e.g., Llama 3.1 8B) on Colab. Accept the specialisation trade-off — the model will be weaker at general Q&A but expert at clause parsing. Deploy locally via Ollama so data never leaves the firm's infrastructure.

A customer-support team wants to replace GPT-4 API calls with a cheaper local model that always responds in their brand's specific tone and escalation format.

Export historical support tickets as input/output pairs (customer message → agent reply in brand format). Fine-tune a small model to reduce per-query cost. Use the Ollama Modelfile SYSTEM prompt to reinforce tone. The fine-tuned small model outperforms a generic large model on this narrow task at a fraction of the cost.

// What mistakes should you avoid when fine-tuning an LLM?

Bad training data produces a bad fine-tuned model — garbage in, garbage out. Do not rush the dataset assembly step.
Skipping the runtime restart after pip install in Colab causes silent import errors that are difficult to debug.
Using too large a base model on a free Colab T4 will either time out or exceed GPU RAM — start small, validate the pipeline, then scale up.
Forgetting to convert output JSON objects to strings (via JSON.dumps) before writing to the dataset causes training to fail or produce malformed outputs.
Not matching the stop token in the Modelfile to the end-of-text token used during training causes the model to hallucinate beyond the intended response boundary.
Downloading the GGUF file before verifying inference inside Colab wastes 20–30 minutes if the model behaviour is wrong.
Fine-tuning a model for a specific domain will make it worse at general tasks — do not use a fine-tuned model as a drop-in replacement for a general assistant.
The max_seq_length in the SFTTrainer must match the value set when loading the base model, or training will error out.
A small dataset (e.g., 500 examples) with a small model will produce inconsistent outputs — more examples always improve reliability.

// What key terms should you know before fine-tuning an LLM?

Fine-tuning: Taking a pre-trained language model and teaching it to be better at a specific task by feeding it domain-specific input/output examples — like hiring an experienced chef and training them on your restaurant's particular recipes rather than teaching someone to cook from scratch.
Parameter Tuning: Adjusting model settings like temperature or top-K to change output behaviour without modifying the model's weights. Analogous to adjusting a car's radio — distinct from fine-tuning, which is like teaching the car to drive in a completely different neighborhood.
Unsloth: An open-source, free Python library that makes fine-tuning LLMs extremely fast and efficient. The primary tool used in this pipeline for loading base models, applying LoRA, and exporting to GGUF.
LoRA (Low-Rank Adaptation): A technique that adds small trainable adapter layers on top of a frozen base model's weights, enabling fast and data-efficient fine-tuning without modifying the entire model.
GGUF: The model file format that Ollama understands and can run. All fine-tuned models must be exported in GGUF format (via Unsloth's save_pretrained_gguf) before they can be loaded into Ollama.
Modelfile: A plain-text configuration file (capital M) that defines a custom model setup for Ollama. It specifies the source GGUF file, inference parameters (temperature, top_p, stop tokens), the prompt template structure, and an optional system message.
SFTTrainer: Supervised Fine-Tuning Trainer — the Hugging Face / Unsloth class that executes the actual fine-tuning process. It takes the model, tokenizer, formatted dataset, and training arguments as inputs.
format_prompt function: A user-written Python function that merges a single input/output training example into one string, including a terminal end-of-text token. Must be customised to match the structure of your specific dataset.
4-bit quantisation (load_in_4bit): A technique that reduces model memory footprint by representing weights in 4-bit precision instead of 16 or 32-bit. Enables large models to fit in limited GPU RAM during fine-tuning.
Ollama: A local model runtime that allows you to run and interact with GGUF-format LLMs on your own machine, including custom fine-tuned models registered via a Modelfile.
T4 GPU: The free GPU runtime tier available in Google Colab (Tesla T4). Sufficient for fine-tuning small models (e.g., Phi-3 Mini) on datasets of a few hundred to low thousands of examples.

// FREQUENTLY ASKED QUESTIONS

What is LLM fine-tuning and how is it different from prompting?

Fine-tuning takes a pre-trained language model and teaches it to be better at a specific task by feeding it domain-specific input/output examples, actually restructuring the model's knowledge. Prompting only changes the instructions you give a fixed model. Fine-tuning is like teaching a car to drive in a new neighborhood, while prompting or parameter tuning is like adjusting the radio.

What is Unsloth and why use it for fine-tuning?

Unsloth is a free, open-source Python library that makes fine-tuning LLMs extremely fast and memory-efficient. In this pipeline it loads the base model, applies LoRA adapters, runs training, and exports the result to GGUF format. It lets you fine-tune small models like Phi-3 Mini on a free Colab T4 GPU in about 10 minutes.

How do I fine-tune an open-source LLM and run it in Ollama?

Assemble a JSON dataset of input/output pairs, open the Unsloth notebook in Google Colab with a T4 GPU, load a base model, apply LoRA adapters, train with SFTTrainer, verify inference, then export to GGUF. Download the .gguf file, write a Modelfile pointing to it, and run 'ollama create' to register and run the model locally.

How do I prepare a training dataset for fine-tuning?

Create a JSON file of input/output pairs, each with an 'input' key and an 'output' key. Aim for hundreds to thousands of examples representing the exact task. Output values must be strings — use JSON.dumps if your output is a JSON object. If real data is scarce, you can use an LLM to generate synthetic examples.

How does fine-tuning compare to just using GPT-4 with a good prompt?

Fine-tuning produces a smaller, cheaper, local model that consistently outputs your exact format, while GPT-4 with prompting stays general but costs per API call and can drift. Fine-tune when you have a narrow, repetitive task, privacy requirements, or high query volume. Use GPT-4 prompting when you need broad general capability or don't have enough training data.

When should I fine-tune a model instead of using parameter tuning?

Fine-tune when you need the model to work with domain-specific data it has never seen, consistently output a specific format, or replace a large expensive model with a smaller specialized one. Use parameter tuning (temperature, top-K) only when you want to nudge an existing model's output style. Fine-tuning restructures knowledge; parameter tuning just adjusts behaviour.

What results can I expect from fine-tuning a small model?

A well-trained small model like Phi-3 Mini can reliably match your exact output format and outperform a generic large model on your narrow task at a fraction of the cost. Expect the model to become worse at general tasks — that's the specialisation trade-off. Reliability scales with dataset size: 500 examples may be inconsistent, thousands are far more reliable.

Do I need a GPU to fine-tune an LLM?

No — Google Colab's free T4 GPU runtime is sufficient for fine-tuning small models like Phi-3 Mini on datasets of a few hundred to low thousands of examples. Local training is only viable if you own a 4080/4090-tier GPU or stronger. For most beginners, Colab is strongly recommended over local hardware.

What is GGUF and why does Ollama need it?

GGUF is the model file format Ollama understands and can run. Fine-tuning frameworks like Unsloth output models in other formats by default, so you must explicitly export to GGUF using save_pretrained_gguf before loading into Ollama. If you skip this step, Ollama cannot load your model.

How long does fine-tuning take on a free Colab T4?

A small model like Phi-3 Mini on roughly 500 examples takes about 10 minutes to train on a T4 GPU. Exporting to GGUF and downloading the file adds another 20–30 minutes depending on model size and internet speed. Larger base models and bigger datasets increase both training and export time significantly.

Why does my fine-tuned model keep generating text past the answer?

Because the stop token in your Ollama Modelfile doesn't match the end-of-text token you used during training. The model needs a consistent terminal token to know where each response ends. Set the Modelfile's stop PARAMETER to the exact same token (e.g., <|endoftext|>) you added in your format_prompt function.

// GET THIS SKILL — FREE