Zen van Riel Local AI Fine-Tuning Pipeline

Last updated: 22 August 2026

Apply a structured, five-step fine-tuning pipeline to transform any open-source base model into a task-specific or persona-specific AI that behaves in ways no prompt or RAG system can reliably replicate.

// TL;DR

The Zen van Riel Local AI Fine-Tuning Pipeline is a structured, five-step process for turning any open-source base model into a task-specific or persona-specific AI using a lightweight LoRA adapter. Use it when prompt engineering and RAG have hit their ceiling — when output still sounds generic, robotic, or off-persona despite iteration. The pipeline runs a decision ladder to confirm fine-tuning is warranted, engineers a clean chat-format dataset, trains a LoRA adapter on Nvidia GPU hardware, evaluates against the vanilla base model, and exports to GGUF for local deployment in LM Studio or Ollama.

Framework

// When should you use the Local AI Fine-Tuning Pipeline?

Use this skill when a user needs to move beyond prompt engineering and RAG because the target behaviour — consistent tone, embedded style, stable domain knowledge — must be baked directly into the model. Trigger it whenever prompt tricks and retrieval have already been tried and the output still sounds generic, robotic, or off-persona.

// What do you need before you start fine-tuning?

Use-case goalrequired
What the fine-tuned model must do that a prompted or RAG-augmented model cannot reliably do (e.g., 'write in my voice every time', 'answer legal queries using stable statute law').
Raw training data sourcerequired
The corpus to train on — transcripts, logs, documentation, blog posts, examples, etc. — and an estimate of token volume.
Target base modelrequired
The open-source base model to fine-tune (e.g., Qwen 3.5 27B). Size determines VRAM requirements and training time.
Available hardwarerequired
GPU vendor (Nvidia preferred, AMD possible, Apple Silicon discouraged), VRAM capacity, and whether GPUs can be paired.
Output deployment target
Where the finished model will run — e.g., LM Studio, Ollama — which determines the required export format.

// What core principles guide effective fine-tuning?

Prompt ceiling principle

Prompt engineering and RAG can guide a model but cannot escape the behaviours baked in during its original training. When output is still robotic or off-style after prompt iteration, you have hit the prompt ceiling and fine-tuning is the only path forward.

LoRA adapter approach

Never retrain billions of parameters. Instead, create a tiny trainable LoRA adapter — targeting roughly 0.5–1.5% of the model's parameters — that injects new behaviour without touching the base weights. This slashes training data requirements, hardware cost, and iteration time.

Garbage in, garbage out — amplified

A language model cannot self-correct dirty training data. Spelling errors, wrong formats, and mismatched prompt-response pairs are baked into the fine-tuned model. Data set engineering is not a shortcut step — it is the step most responsible for final model quality.

Decision ladder before fine-tuning

Always exhaust cheaper options first: (1) better prompt, (2) RAG, (3) agentic loop. Only proceed to fine-tuning if all three fail to meet the performance bar, or if owning an in-house model is itself a strategic advantage over competitors who are only prompt-injecting.

Chat-format correspondence

Fine-tuning data must mirror the chat format the model will be used in. Raw documents (transcripts, docs) are not usable as-is; they must be transformed into prompt-response pairs that match the conversational structure the model expects at inference time.

Hardware reality check

VRAM is the hard constraint. A model fitting in VRAM does not mean it fine-tunes fast; RAM offloading grinds fine-tuning to a halt. Nvidia + CUDA remains the first-class citizen; AMD + ROCm is viable for hobbyists; Apple Silicon (MLX) is effectively a red-zone choice for fine-tuning.

// How do you fine-tune a local model step by step?

1
Run the decision ladder to confirm fine-tuning is warranted
Ask: Does a better system prompt solve the problem? If yes, stop. If no, add RAG. If RAG is insufficient, try an agentic loop. Only if all three fail — or if the strategic goal is an in-house model — proceed to fine-tuning. Document why each prior tier failed; this evidence guides your evaluation criteria later.
2
Collect and size your raw training data
Gather the corpus (transcripts, logs, docs, examples). Count raw tokens. For an 8B parameter model, target 1–2 million+ tokens of raw data before transformation. Identify data quality issues up front — automatic transcripts, OCR errors, inconsistent formatting — so you know the cleaning scope before committing to the pipeline.
3
Engineer the data set — clean, format, and generate prompt-response pairs
Clean all noise: spelling errors, malformed punctuation, artefacts from automatic transcription. Then transform raw documents into chat-format prompt-response pairs. Use a local language model to auto-generate plausible questions for each text snippet, then pair the snippet as the answer. Tailor pair structure to the end use case: if the goal is blog writing, prompts should always follow a consistent template (e.g., 'Write a blog post given this news summary: {paste}') so the formatting rules become embedded in the model. Do not skip or rush this step — poor data set engineering is the single biggest cause of a poor fine-tuned model.
4
Run LoRA training on target hardware
Use a LoRA adapter targeting 0.5–1.5% of the model's parameters. Prefer Nvidia GPU with CUDA for compatibility and speed; AMD + ROCm is a viable fallback with variable results; avoid Apple Silicon for fine-tuning. Ensure the model fits in dedicated VRAM — do not rely on RAM offloading. For a 27B parameter model expect 14+ GB VRAM. Budget 2–3 hours per training run on a high-end consumer GPU (e.g., RTX 5090) for mid-to-large models; set parameters carefully because wrong settings waste multi-hour runs. Disable behaviours you don't want (e.g., extended chain-of-thought 'thinking' mode) explicitly during this phase.
5
Evaluate against the vanilla base model
Write a suite of test prompts representative of the real use case. Run identical prompts on both the vanilla base model and your fine-tuned model. Compare on the specific criteria that motivated fine-tuning (tone, brevity, accuracy, format). Use evaluation results to diagnose failures: wrong output usually traces back to data set engineering errors or LoRA parameter misconfiguration. Iterate on data or parameters, not on post-hoc prompting of the fine-tuned model.
6
Export to GGUF format for deployment
Convert the merged model (base + LoRA adapter) to GGUF, the consumer-friendly format compatible with LM Studio, Ollama, and similar local inference platforms. Verify the exported model runs correctly on the intended deployment environment before declaring the pipeline complete.

// What does the fine-tuning pipeline look like in real scenarios?

A consultant wants an AI assistant that answers client questions in their established communication style — direct, jargon-free, concise — rather than producing verbose, generic AI-sounding responses.

Collect all existing written outputs (emails, reports, proposals) as the corpus. Clean and transform them into prompt-response pairs using a local model to generate plausible client questions matched to each passage. Run LoRA training on the target base model. Evaluate by sending the same client-style questions to both the vanilla model and the fine-tuned model and scoring for stylistic match. Export to GGUF for use in a local inference tool.

A legal tech team needs a model that reliably answers questions about a stable body of statute law without hallucinating, while still handling recent amendments.

Fine-tune the base model on the stable statutory corpus (laws unlikely to change) so that knowledge is baked in. Layer RAG on top for recent amendments or case law updates. This hybrid approach bakes stable knowledge into the model for reliability and speed, while keeping dynamic knowledge retrievable without retraining.

A content team wants an AI that always produces blog posts in their house style — specific heading structure, sentence length, and vocabulary — without needing a multi-paragraph style guide injected into every prompt.

Collect existing on-brand blog posts as the corpus. Engineer prompt-response pairs where every prompt follows the template 'Write a blog post on this topic: {topic}' and every response is a correctly formatted example post. Fine-tune so the formatting rules are embedded rather than prompted. Evaluate by requesting posts with minimal prompting and checking for unprompted adherence to house style.

// What mistakes should you avoid when fine-tuning?

Skipping the decision ladder and jumping straight to fine-tuning when a better prompt or RAG would have solved the problem — fine-tuning costs at minimum a full weekend and significant hardware resources.
Feeding raw, uncleaned data (e.g., auto-generated transcripts with errors) directly into training — the model cannot self-correct bad input and will bake those errors in permanently.
Failing to convert raw documents into chat-format prompt-response pairs — training data that doesn't match the conversational structure the model uses at inference time produces a broken fine-tuned model.
Relying on RAM offloading to compensate for insufficient VRAM — this grinds fine-tuning to a halt and often fails entirely; dedicated GPU VRAM is a hard requirement.
Choosing Apple Silicon for fine-tuning — MLX-format model ports are rare, fine-tuning is slow on silicon chips, and the ecosystem is not expected to close the gap in the short term.
Setting LoRA training parameters incorrectly and only discovering the mistake after a 2–3 hour training run — always validate data and parameters on a short test run before committing to a full training job.
Skipping a structured evaluation pipeline — without explicit tests comparing the fine-tuned model to the vanilla base, you cannot diagnose whether failures originate in data set engineering or LoRA configuration.
Assuming that because a model fits in VRAM it will fine-tune at an acceptable speed — model fit and fine-tuning throughput are separate concerns; benchmark before committing hardware.

// What key terms should you know about local AI fine-tuning?

LoRA adapter: Low Rank Adaptation — a tiny trainable adapter layer created from your own training data that is injected into a base model without retraining all its parameters. Targets roughly 0.5–1.5% of total model parameters, dramatically reducing data, time, and hardware requirements.
Base language model: The pre-trained open-source model (e.g., Qwen 3.5 27B) that serves as the starting point before any fine-tuning adapter is applied.
Fine-tuned model: The base language model after the LoRA adapter has been merged in, producing a new model with behaviour, style, or knowledge baked directly into its weights.
RAG (Retrieval Augmented Generation): A paradigm where a model retrieves relevant documents from a structured database (vector or otherwise) and self-injects them into its prompt at inference time, allowing up-to-date or private knowledge without retraining.
Prompt ceiling: The performance limit of pure prompt engineering — the point at which further prompt iteration cannot overcome the base model's trained behaviours, making fine-tuning the only viable path to the desired output.
Data set engineering: The process of cleaning raw training data and transforming it into chat-format prompt-response pairs suitable for LoRA fine-tuning. The most failure-prone and most consequential step in the pipeline.
Chat-format prompt-response pairs: Training examples structured as a user question paired with the desired model answer, matching the conversational format the model will use at inference time.
GGUF export: A consumer-friendly model file format produced after fine-tuning that enables the merged model to run on local inference platforms such as LM Studio and Ollama.
CUDA: Nvidia's software development kit that allows Python libraries and fine-tuning frameworks to interact efficiently with Nvidia GPUs. The primary reason Nvidia is the first-class citizen for fine-tuning.
ROCm: AMD's counterpart to CUDA, enabling fine-tuning on AMD GPUs. Functional but not first-class — compatibility and performance vary, making it a viable hobbyist option but not the recommended path.
MLX format: The model format required to run or fine-tune on Apple Silicon (M-series chips). Rarely ported by model developers, making Apple Silicon an effectively red-zone choice for fine-tuning.
Agentic loop: An architecture where the language model itself decides whether to pull in more information, invoke tools, or delegate to another agent — a step above RAG in complexity and the final option to try before resorting to fine-tuning.
VRAM (Video RAM): The dedicated memory on a GPU. The hard constraint for fine-tuning — the model's parameters and LoRA adapter must fit in VRAM; RAM offloading is not a viable substitute.

// FREQUENTLY ASKED QUESTIONS

What is the Zen van Riel Local AI Fine-Tuning Pipeline?

It's a five-step framework for transforming an open-source base model into a task- or persona-specific AI using a small LoRA adapter. The steps are: run a decision ladder, collect and size raw data, engineer a clean chat-format dataset, train the LoRA adapter on GPU, evaluate against the vanilla base model, and export to GGUF. It bakes behaviour into the model that no prompt or RAG system can reliably replicate.

What is a LoRA adapter and why does fine-tuning use it?

A LoRA (Low Rank Adaptation) adapter is a tiny trainable layer injected into a base model that targets roughly 0.5–1.5% of its parameters instead of retraining billions of weights. It's used because full retraining is prohibitively expensive; LoRA slashes training data requirements, hardware cost, and iteration time while still baking new tone, style, or knowledge directly into the model.

When should I fine-tune instead of using prompts or RAG?

Fine-tune only after exhausting the decision ladder: a better prompt, then RAG, then an agentic loop. If all three fail to meet your performance bar — output still sounds generic or off-persona — you've hit the prompt ceiling and fine-tuning is the only path. Also fine-tune if owning an in-house model is itself a strategic advantage over competitors who only prompt-inject.

How do I prepare training data for fine-tuning?

Clean all noise first — spelling errors, malformed punctuation, transcription artefacts — because the model cannot self-correct dirty data and bakes errors in permanently. Then transform raw documents into chat-format prompt-response pairs that mirror how the model is used at inference. Use a local model to auto-generate plausible questions for each text snippet, pairing the snippet as the answer, and follow consistent prompt templates.

How much data and hardware do I need to fine-tune a local model?

For an 8B model, target 1–2 million+ raw tokens before transformation. Hardware-wise, VRAM is the hard constraint: a 27B model needs 14+ GB VRAM, and the model must fit in dedicated VRAM — RAM offloading grinds training to a halt. Prefer Nvidia GPU with CUDA; budget 2–3 hours per training run on a high-end consumer GPU like an RTX 5090.

How does fine-tuning compare to RAG?

RAG retrieves documents at inference time and injects them into the prompt, making it ideal for dynamic or frequently changing knowledge without retraining. Fine-tuning bakes behaviour and stable knowledge directly into model weights, delivering consistent tone, style, and reliability that RAG cannot guarantee. They're complementary: fine-tune stable knowledge and persona, then layer RAG on top for recent updates like new statutes or case law.

Can I fine-tune on Apple Silicon (M-series Mac)?

It's discouraged — effectively a red-zone choice. Apple Silicon requires MLX-format model ports, which developers rarely release, and fine-tuning is slow on silicon chips. The ecosystem is not expected to close this gap in the short term. Nvidia + CUDA remains the first-class citizen; AMD + ROCm is a viable hobbyist fallback with variable results.

What results can I expect from fine-tuning a local model?

A model that consistently produces your desired tone, style, or format without a multi-paragraph prompt — for example, blog posts in your house style or client answers in your voice every time. Behaviour becomes embedded rather than prompted. Evaluation should show clear improvement over the vanilla base on your specific criteria (tone, brevity, accuracy, format) when given identical test prompts.

Why does my fine-tuned model produce bad output?

Bad output usually traces back to two root causes: dataset engineering errors or LoRA parameter misconfiguration. If training data wasn't cleaned or wasn't converted to chat-format prompt-response pairs, errors get baked in permanently. If LoRA settings were wrong, a full training run can be wasted. Diagnose by comparing against the vanilla base model — never fix it with post-hoc prompting.

What format do I export the fine-tuned model in?

Convert the merged model (base + LoRA adapter) to GGUF, the consumer-friendly format compatible with local inference platforms like LM Studio and Ollama. Always verify the exported model runs correctly on your intended deployment environment before declaring the pipeline complete.

How long does the full fine-tuning process take?

At minimum a full weekend. Dataset engineering — cleaning and building chat-format pairs — is the most time-consuming and consequential step. Each LoRA training run on a high-end consumer GPU takes 2–3 hours for mid-to-large models, and wrong parameters waste that time entirely, so validate on a short test run first before committing to a full job.

// GET THIS SKILL — FREE