Savvita LLM Fine-Tuning Pipeline Framework

// When should you use the Savvita LLM Fine-Tuning Pipeline Framework?

Use this skill whenever you need to adapt a pre-trained large language model to a specific domain, task, or alignment requirement, or when you need to decide which fine-tuning stage and technique best matches your data, compute, and output goals.

// What do you need before starting an LLM fine-tuning project?

• Target domain or use caserequired
  What industry, task, or capability the model needs to serve (e.g., pharma QA, legal chatbot, code assistant).
• Data type and formatrequired
  Whether the available data is plain text (PDF, TXT), instruction/QA pairs, or preference-ranked responses (chosen/rejected).
• Starting modelrequired
  The base or instruct LLM to be fine-tuned (e.g., LLaMA 3, Mistral 7B, TinyLlama). Identify whether it is a base model or already instruction-tuned.
• Compute constraintsrequired
  Available GPU memory, number of GPUs, and whether a free or paid runtime is available (e.g., single T4, A100, multi-GPU cluster).
• Desired output behaviourrequired
  Whether the end goal is domain knowledge injection, conversational ability, structured QA, or human-aligned safe responses.

// What core principles guide stage-correct LLM fine-tuning?

// How do you fine-tune an LLM step by step with this framework?

1. 1

   Identify which training stage you are entering

   Map your goal to one of the three stages: (1) Unsupervised Pre-Training — build a base model from scratch (almost never done outside large organisations with massive infrastructure); (2) SFT — adapt or instruction-tune an existing base model; (3) Preference-Based Alignment — align an already SFT-tuned model to human feedback. Most practitioners start at Stage 2 or 3.

2. 2

   Select and audit your base model

   Pick a model from Hugging Face matching your compute constraints (smaller models like TinyLlama 1.1B for single free GPU; larger models like LLaMA 3 8B for paid/multi-GPU). Confirm it is a base model if you intend to run the full SFT pipeline. Check the model card for context window size — this determines your chunking strategy.

3. 3

   Audit your data and assign it to a data-level category

   Classify your data as: (a) Plain text (PDF, TXT, web crawl) → Non-Instructional Finetuning; (b) Instruction-and-response or QA pairs → Instructional Finetuning; (c) Prompt + chosen + rejected response triples → Preference Alignment. If you have plain text but want a chatbot, you must pass through non-instructional finetuning first, then create or source instructional data for the next stage.

4. 4

   Choose your parameter-level strategy (PEFT technique)

   For single GPU or limited memory: use LoRA or QLoRA (QLoRA when loading a quantised model). For experimental or research settings: consider DoRA, IA3, adapter layers, or prefix tuning. Avoid full fine-tuning unless you have multi-GPU infrastructure and very large domain datasets. Set LoRA hyperparameters: rank (r), alpha, target modules.

5. 5

   Prepare and preprocess the dataset for your chosen stage

   Non-Instructional: Extract plain text from source (use pymupdf/fitz for PDFs), clean and chunk by paragraph, semantic boundary, or token length (respecting the model's context window). Convert to Hugging Face Dataset format with a single 'text' column. Set labels = input_ids (for causal language modelling / next-token prediction). Instructional: Format as instruction + response pairs; apply the model's chat template. Preference Alignment: Ensure dataset has 'prompt', 'chosen', and 'rejected' columns.

6. 6

   Configure tokenisation with truncation, padding, and max-length

   Always set truncation=True and padding=True with a fixed max_length matching your chunk strategy and model context window. Assign pad_token = eos_token if pad_token is None. For causal LM, set labels = input_ids so the model trains on next-token prediction across the entire sequence.

7. 7

   Install and configure the correct framework and libraries

   Core stack: transformers, datasets, accelerate (for multi-GPU; install even on single GPU for dependencies), bitsandbytes (for quantised/QLoRA loading), peft (for LoRA configuration and PEFT adapters), trl (Transformer Reinforcement Learning — required for SFT Trainer and DPO Trainer). For PDF loading: pymupdf (fitz). Optional high-performance frameworks: Unsloth (faster LoRA training), Axolotl, LLaMA Factory — all produce compatible outputs.

8. 8

   Load the model and apply the PEFT configuration

   Load the base model using AutoModelForCausalLM. If using QLoRA, load with BitsAndBytesConfig (lower precision / 4-bit or 8-bit). Apply LoraConfig via the peft library, specifying rank, alpha, dropout, and target modules. Wrap the model with get_peft_model().

9. 9

   Define training arguments and launch training

   Use TrainingArguments (from transformers) or SFTTrainer (from trl) for instructional finetuning; use DPOTrainer (from trl) for preference alignment. Key hyperparameters: output_dir, num_train_epochs, per_device_train_batch_size, learning_rate, logging_steps, save_steps. Use DataCollatorForLanguageModeling for non-instructional finetuning. Run training and monitor loss.

10. 10

    Sequence multiple fine-tuning stages if needed

    The recommended production sequence for a domain-specific conversational model: (1) Start with base model → Non-Instructional Finetuning on domain plain text → domain-expert base model. (2) Continue with Instructional Finetuning on domain QA pairs → domain chatbot. (3) Apply DPO on chosen/rejected pairs → aligned, safe, helpful domain chatbot. Each stage takes the output model of the previous stage as its starting point.

11. 11

    Validate and iterate

    After each stage, test the model with representative prompts. Non-instructional output will be continuous text, not structured answers — this is expected. Only after instructional finetuning should the model follow instructions. Only after preference alignment should the model be consistently polite, safe, and human-value-aligned.

// What do real LLM fine-tuning projects look like using this framework?

// What mistakes should you avoid when fine-tuning an LLM?

• Confusing a base model with an instruct/chat model — always check the model name and card before choosing your starting stage; using an already-instruction-tuned model as if it were a base model wastes compute and corrupts the pipeline.
• Skipping the non-instructional finetuning stage when domain knowledge is critical — jumping directly to instructional finetuning on QA pairs without first injecting domain knowledge produces a chatty model with shallow domain understanding.
• Attempting full fine-tuning on a single consumer GPU — full fine-tuning trains all weights and biases and requires huge GPU memory plus multi-GPU setup; always use LoRA or QLoRA instead.
• Using the wrong data format for the stage — plain text fed into an instructional finetuning pipeline, or QA pairs fed into a non-instructional pipeline, will produce a model that does not behave as expected.
• Not setting pad_token = eos_token when the tokeniser has no pad token — this causes padding errors during batched training.
• Ignoring chunk size relative to the model's context window — chunks longer than the model's max token limit will be silently truncated, losing information; chunks that are too short are inefficient.
• Treating DPO/RLHF as optional polish — if the end product is a user-facing assistant, skipping preference alignment produces a model that may be technically capable but unsafe, impolite, or misaligned with user expectations.
• Assuming one framework fits all — Hugging Face Transformers, Unsloth, Axolotl, and LLaMA Factory have different performance profiles and trade-offs; benchmark on your specific model and hardware before committing to one.

// What are the key terms in LLM fine-tuning explained?

Unsupervised Pre-Training

The first stage of the LLM training pipeline, also called self-supervised learning, where the model is trained on massive internet-scale text to predict the next token. This stage develops the model's general language understanding and produces a base model.

SFT (Supervised Fine-Tuning)

The second stage of the LLM training pipeline, encompassing both non-instructional and instructional finetuning. Operates on two axes: parameter level (full vs. PEFT) and data level (plain text vs. instruction-response pairs).

Non-Instructional Finetuning

SFT using plain text data (PDF, TXT, web crawl) formatted for next-token prediction. The goal is domain adoption — injecting domain-specific knowledge into the model — not teaching conversational or instruction-following behaviour.

Instructional Finetuning

SFT using data formatted as instruction-and-response (or input-output, question-answer) pairs. The goal is to make the model chat-enabled and capable of following human instructions — converting a base model into a conversational AI or chatbot.

Preference-Based Alignment

The third stage of the LLM training pipeline, also called alignment with human feedback. Retrains an SFT model on human-ranked response pairs (chosen/rejected) to make it polite, safe, helpful, and aligned with human values.

RLHF (Reinforcement Learning from Human Feedback)

A preference alignment technique that uses the PPO (Proximal Policy Optimization) reinforcement learning algorithm to align model responses with human preferences. Used by OpenAI for ChatGPT.

DPO (Direct Preference Optimization)

A supervised preference alignment technique that trains directly on chosen/rejected response pairs without requiring a separate reward model or reinforcement learning loop. The currently preferred technique for preference alignment.

PPO (Proximal Policy Optimization)

The reinforcement learning algorithm underlying RLHF. Stands for Proximal Policy Optimization.

PEFT (Parameter Efficient Fine-Tuning)

A family of techniques that train only a subset of model parameters (not all weights and biases), enabling fine-tuning on a single GPU with smaller memory. Umbrella term for LoRA, QLoRA, DoRA, adapter layers, IA3, bitfit, and prefix tuning.

Full Fine-Tuning

Training all parameters (weights and biases) of a model. Requires huge GPU memory and typically a multi-GPU setup. Generally avoided in favour of PEFT techniques.

LoRA (Low-Rank Adaptation)

The foundational PEFT technique. Inserts trainable low-rank decomposition matrices into the transformer's existing layers instead of updating all weights. The base for most other PEFT variants.

QLoRA (Quantised LoRA)

LoRA applied on top of a quantised (lower-precision) model. Enables memory-efficient loading and training of large models on limited hardware. The Q represents quantisation.

DoRA (Weight Decomposition Low-Rank Adaptation)

A PEFT technique that decomposes weight matrices before applying low-rank adaptation. An extension of LoRA.

IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations)

A PEFT technique that improves upon LoRA by infusing learned vectors that inhibit or amplify inner activations of the transformer. Part of the Hugging Face PEFT library.

Adapter Layers

A PEFT technique that appends small trainable layers to existing transformer blocks, inspired by LoRA.

Prefix Tuning

A PEFT technique that prepends trainable prefix tokens to the input sequence, conditioning the model's behaviour without modifying its weights.

Base Model

A model that has undergone only unsupervised pre-training. It has general language and world knowledge but cannot reliably follow instructions or hold conversations. Identified in model names by the absence of 'instruct', 'chat', or similar suffixes.

Next Token Prediction

The core training objective of both unsupervised pre-training and non-instructional finetuning. The model learns to predict the next token given all previous tokens in a sequence. Also called causal language modelling.

TRL (Transformer Reinforcement Learning)

A Hugging Face library providing SFTTrainer and DPOTrainer, required for implementing both instructional finetuning and preference alignment in the Hugging Face ecosystem.

bitsandbytes

A library (by Tim Dettmers, hosted under Hugging Face) used to load quantised models in lower precision (4-bit or 8-bit) for memory-efficient training, enabling QLoRA.

Chosen / Rejected

The two mandatory columns in every preference alignment dataset. 'Chosen' is the response rated positively by humans; 'rejected' is the response rated negatively. DPO trains the model to prefer chosen over rejected responses.

// FREQUENTLY ASKED QUESTIONS