How to Fine-Tune an LLM for Structured Data Extraction
For Backend developers automating data extraction · Based on Tech With Tim LLM Fine-Tuning to Ollama Pipeline
// TL;DR
Backend developers can fine-tune a small open-source model like Phi-3 Mini to extract structured JSON (product names, prices, categories) from raw HTML or unstructured text with near-perfect format consistency. Instead of wrangling brittle regex or paying per-token for GPT-4, you train once on input/output pairs, export to GGUF, and run the model locally via Ollama. Use this when you have a repetitive extraction task, need deterministic output shape, and want to eliminate API costs and latency in your data pipeline.
Why fine-tune instead of parsing HTML with rules?
Rule-based extraction breaks the moment a page's markup changes, and prompting GPT-4 for structured output works but drifts, costs money per call, and adds network latency. Fine-tuning a small model teaches it your exact extraction task so it returns consistent JSON every time. As the pipeline's core principle states, fine-tuning restructures the model's knowledge — it's like teaching your car to drive in a completely different neighborhood, not just adjusting the radio.
For a developer extracting product data (name, price, category, manufacturer) from HTML snippets, the payoff is a local model that reliably emits the same JSON schema for any new input, with no API bill.
How do I build the training dataset?
Assemble a JSON file of input/output pairs. Inputs are your raw HTML fragments; outputs are the corresponding structured records. Critically, convert each output object to a string with `JSON.dumps` before writing it — outputs must be strings or training fails. Aim for 500 or more examples; 500 is a viable starting point but thousands produce more reliable output.
If you lack real labelled data, use a larger LLM to generate synthetic HTML-to-JSON pairs from a template, then review them for quality. Remember: data quality is the ceiling of your fine-tuned model.
```
{"input": "
```
How do I train and deploy it?
Open the Unsloth notebook in Google Colab, switch the runtime to a T4 GPU, and upload your dataset. Install Unsloth and — this is mandatory — restart the session after the pip install cell or imports fail silently. Load Phi-3 Mini via `FastLanguageModel` with `load_in_4bit=True` and set `max_seq_length` to cover your longest HTML fragment (e.g., 2048).
Write a `format_prompt` function that merges each input and output into one string ending in an end-of-text token like `<|endoftext|>`. Apply LoRA adapters with `get_peft_model`, then run the `SFTTrainer`. Training ~500 examples on Phi-3 Mini takes about 10 minutes. Watch the loss decrease to confirm learning.
Before downloading anything, run inference inside Colab on several real HTML snippets and confirm the JSON is well-formed. Only then export with `save_pretrained_gguf` and download the `.gguf` file — re-downloading after finding a bug wastes 20–30 minutes.
How do I wire it into Ollama?
In the folder with your `.gguf`, create a `Modelfile` (capital M) that points `FROM ./yourfile.gguf`, sets a low temperature for deterministic output, and — most importantly — sets the `stop` parameter to the exact same `<|endoftext|>` token you used in training. Mismatched stop tokens cause the model to hallucinate past your JSON. Add a TEMPLATE block and a SYSTEM message. Then run `ollama create product-extractor -f Modelfile`, verify with `ollama list`, and test with `ollama run product-extractor`.
You now have a local extraction microservice you can call from your backend with zero per-request cost.
Next step: Collect 500 real HTML/JSON pairs from your existing pipeline logs, run them through the Unsloth Colab notebook, and benchmark the fine-tuned Phi-3 Mini against your current extraction method.
// FREQUENTLY ASKED QUESTIONS
Which base model is best for JSON extraction?
Start with Phi-3 Mini. It trains fast on a free Colab T4 and is more than capable of learning a fixed JSON extraction schema. Validate your pipeline and data with it first, then only upgrade to Llama 3.1 8B if the small model can't capture your task's complexity after you've maximised dataset quality.
How do I stop the model from adding extra text around the JSON?
Set a low temperature in your Modelfile and make sure the stop token matches the end-of-text token from training. Also ensure every training output was a clean JSON string with no surrounding prose. The model learns exactly what you show it, so consistent, prose-free outputs during training produce clean JSON at inference.
Can I run this extraction model in production?
Yes. Once registered in Ollama, you can call the model locally via Ollama's API from your backend, giving you deterministic structured output with no external API cost or latency. For higher throughput, run it on a machine with a capable GPU. Keep a general model separate — the fine-tuned one is specialised only for extraction.