Frequently Asked Questions About Savvita LLM Fine-Tuning Pipeline Framework
22 answers covering everything from basics to advanced usage.
// Basics
What does PEFT stand for and what techniques does it include?
PEFT stands for Parameter Efficient Fine-Tuning — a family of techniques that train only a subset of model parameters rather than all weights and biases, enabling fine-tuning on a single GPU. It's an umbrella term covering LoRA, QLoRA, DoRA, IA3, adapter layers, bitfit, and prefix tuning. LoRA is the foundational technique the others extend.
What is next-token prediction and where is it used?
Next-token prediction, also called causal language modelling, is the objective where the model learns to predict the next token given all previous tokens. It's used in both unsupervised pre-training and non-instructional finetuning. For this objective you set labels = input_ids so the model trains on predicting each subsequent token across the sequence.
What is QLoRA and when should I use it instead of LoRA?
QLoRA is LoRA applied on top of a quantised (4-bit or 8-bit) model, enabling memory-efficient training of large models on limited hardware. Use QLoRA when you need to load a large model that won't fit in full precision on your GPU — load with BitsAndBytesConfig, then apply your LoraConfig. Use plain LoRA when the model fits in full or half precision.
What are the two axes I must think about during SFT?
SFT decisions happen on two independent axes simultaneously: the parameter level (full fine-tuning vs. PEFT techniques like LoRA, QLoRA, DoRA) and the data level (non-instructional plain text vs. instructional instruction/response pairs). Conflating these axes leads to mismatched pipelines. You choose one option from each axis for every SFT run.
// How To
How do I prepare plain text data for non-instructional finetuning?
Extract text from your source (use pymupdf/fitz for PDFs), clean it, and chunk it by paragraph, semantic boundary, or token length respecting the model's context window. Convert to a Hugging Face Dataset with a single 'text' column. Tokenise with truncation and padding at a fixed max_length, then set labels = input_ids for causal language modelling.
How do I format data for preference alignment with DPO?
Structure your dataset with three columns: 'prompt', 'chosen', and 'rejected'. The 'chosen' column holds the human-preferred response; 'rejected' holds the disfavoured one. Feed this to trl's DPOTrainer, which trains the model to prefer chosen over rejected responses. Collect these pairs from human raters or generate and rank candidate responses.
How do I sequence multiple fine-tuning stages for a domain chatbot?
Use the recommended production sequence: (1) base model → non-instructional finetuning on domain plain text → domain-expert base model; (2) → instructional finetuning on domain QA pairs → domain chatbot; (3) → DPO on chosen/rejected pairs → aligned, safe chatbot. Each stage takes the output model of the previous stage as its starting point.
How do I configure tokenisation correctly for training?
Set truncation=True and padding=True with a fixed max_length that matches your chunk strategy and the model's context window. If the tokeniser has no pad_token, assign pad_token = eos_token to avoid padding errors during batched training. For causal LM, set labels = input_ids so the model trains on next-token prediction across the full sequence.
// Troubleshooting
Why does my fine-tuned model produce continuous text instead of answering questions?
This is expected if you only ran non-instructional finetuning — that stage injects domain knowledge via next-token prediction but does not teach instruction-following. To get structured answers, run instructional finetuning next, formatting data as instruction/response pairs and applying the model's chat template. Only after that stage will the model reliably follow instructions.
Why am I getting padding errors during batched training?
Most likely your tokeniser has no pad_token defined. Set pad_token = eos_token before tokenising. Also confirm you enabled padding=True with a consistent max_length, and use DataCollatorForLanguageModeling for non-instructional finetuning so batches are padded uniformly. Mismatched sequence lengths without a pad token break the collation step.
My model runs out of GPU memory during training — what should I do?
Switch to QLoRA: load the model quantised in 4-bit via BitsAndBytesConfig, then apply LoraConfig. Reduce per_device_train_batch_size and use gradient accumulation. Choose a smaller base model (TinyLlama 1.1B or Mistral 7B) and shorten your max_length. Avoid full fine-tuning entirely on a single consumer GPU — it trains all weights and needs multi-GPU memory.
Why does my model lack deep domain knowledge despite instruction tuning?
You likely skipped non-instructional finetuning. Jumping straight to instructional finetuning on QA pairs produces a chatty model with shallow domain understanding. Fix this by first running non-instructional finetuning on your domain plain-text corpus to inject knowledge, then continue with instructional finetuning on the resulting domain-expert model.
// Comparisons
How does DPO compare to RLHF for preference alignment?
RLHF uses PPO reinforcement learning with a separate reward model, making it powerful but complex and unstable to train — it's what OpenAI used for ChatGPT. DPO trains directly on chosen/rejected pairs with a supervised objective, requiring no reward model or RL loop. DPO is simpler, more stable, and currently the preferred technique for most practitioners.
How does fine-tuning compare to RAG for domain knowledge?
Fine-tuning bakes domain knowledge and behaviour into the model's weights, producing consistent tone and style but requiring retraining to update. RAG retrieves external documents at inference, staying current without retraining but adding latency and depending on retrieval quality. This framework focuses on fine-tuning; many production systems combine fine-tuned behaviour with RAG for fresh facts.
How does starting from a base model compare to starting from an instruct model?
A base model has only unsupervised pre-training — general knowledge but no instruction-following — so it's the correct entry for the full three-stage pipeline. An instruct/chat model already went through SFT and possibly alignment, so you'd typically only add lightweight domain SFT or DPO. Using an instruct model as if it were a base model wastes compute and corrupts the pipeline.
How does Unsloth compare to plain Hugging Face for LoRA training?
Unsloth is a high-performance framework offering faster LoRA training and lower memory use than plain Hugging Face Transformers, and it produces compatible outputs. Axolotl and LLaMA Factory are similar alternatives with different trade-offs. Don't assume one framework fits all — benchmark on your specific model and hardware before committing, since performance profiles vary.
// Advanced
What LoRA hyperparameters matter most and how do I set them?
The key LoRA hyperparameters are rank (r), alpha, dropout, and target modules. Rank controls the capacity of the adapter (higher r = more trainable parameters); alpha scales the adapter's contribution; target modules specify which layers (often attention projection layers) get adapters. Start with modest rank values, tune alpha proportionally, and target attention modules unless your task needs broader coverage.
When would I choose DoRA or IA3 over LoRA?
Choose DoRA or IA3 in experimental or research settings where you want to push adaptation quality beyond standard LoRA. DoRA decomposes weight matrices before applying low-rank adaptation; IA3 infuses learned vectors that inhibit or amplify inner activations. Both extend LoRA and live in the Hugging Face PEFT library. For most production work, start with LoRA or QLoRA first.
Can I skip non-instructional finetuning entirely?
Yes, when the base model already has sufficient knowledge for your task. For example, an open-source model may have enough general coding knowledge for a coding assistant, so you'd skip domain injection and go straight to instructional finetuning. Skip non-instructional finetuning only when domain knowledge is not the bottleneck — otherwise you'll get shallow domain understanding.
How do I decide chunk size relative to context window?
Set chunk size to fully use, but not exceed, the model's context window (check the model card). Chunks longer than max token limit are silently truncated, losing information; chunks that are too short waste compute and fragment context. Chunk by semantic or paragraph boundaries where possible, then confirm token counts fit within your fixed max_length.
Should I train from scratch or fine-tune an existing model?
Almost always fine-tune. Training from scratch (unsupervised pre-training) requires internet-scale data and multi-GPU clusters — feasible only for organisations like Meta, Google, or Mistral. For any company without that infrastructure, pick an open-weight base model (LLaMA, Mistral, DeepSeek) and enter at Stage 2 (SFT) or Stage 3 (alignment) depending on whether the model is already instruction-tuned.
Which trl trainers do I use for each stage?
Use SFTTrainer from trl for instructional finetuning, and DPOTrainer from trl for preference alignment. For non-instructional finetuning, use TrainingArguments from transformers with DataCollatorForLanguageModeling. All require you to first load the model with AutoModelForCausalLM and apply LoraConfig via get_peft_model() when using PEFT.