Frequently Asked Questions About AI Anytime LLM Fine-Tuning End-to-End Framework
23 answers covering everything from basics to advanced usage.
// Basics
What is the difference between pre-training and fine-tuning?
Pre-training teaches a model general language knowledge on terabytes of unlabeled text using objectives like next-word prediction, producing broad understanding but no task-specific skill. Fine-tuning specialises that pre-trained model on a much smaller labeled dataset — typically instruction-response pairs — to master a specific task. Fine-tuning is faster and cheaper because it builds on knowledge the model already has.
What is Axolotl and why use it for fine-tuning?
Axolotl is a low-code fine-tuning tool that accepts a single YAML configuration file and handles the entire pipeline: data loading, tokenisation, LoRA/QLoRA setup, training, and checkpointing. It removes boilerplate so you can focus on hyperparameters. This framework recommends it for productivity, but you still need to understand LoRA mechanics, tokenisation, and gradient descent to debug errors and tune performance.
What are LoRA target modules and which ones should I set?
Target modules are the weight matrices LoRA adapts. At minimum, target q_proj (query projection) and v_proj (value projection) in the attention mechanism. For richer adaptation, add k_proj, o_proj, gate_proj, down_proj, and up_proj. If your dataset uses custom vocabulary or syntax, also target embed_tokens and lm_head. Always confirm exact layer names per model family via model.state_dict().keys().
What does gradient accumulation do and when should I use it?
Gradient accumulation simulates a larger effective batch size without the memory cost by splitting a global batch into sequential mini-batches, accumulating their gradients, and performing one weight update. Use it when GPU VRAM constrains your batch size. Effective batch size = micro_batch_size × gradient_accumulation_steps × num_GPUs. If you hit OOM errors, set micro_batch_size to 1 and raise gradient_accumulation_steps to 8 or 16.
// How To
How do I set up Axolotl for fine-tuning?
Clone the Axolotl GitHub repo into your JupyterLab environment, then run pip install packaging followed by pip install -e '.[flash-attn,deepspeed]'. Ensure flash-attention is version 2.0 or higher — version 1.x causes errors. Then navigate to examples/<your-model>/qlora.yml, configure your base model, dataset path, and LoRA hyperparameters, and launch with accelerate launch -m axolotl.cli.train.
How do I configure QLoRA settings in the YAML file?
Set load_in_4bit: true, bnb_4bit_quant_type: nf4 (4-bit NormalFloat, optimal for normally distributed weights), and bnb_4bit_use_double_quant: true to further reduce memory by quantising the quantisation constants. The bits-and-bytes library handles on-the-fly near-lossless quantisation. Expect minor performance degradation as an acceptable tradeoff for being able to run on consumer GPUs.
How do I prepare a dataset for instruction tuning?
Format your data as instruction-response pairs in JSONL or CSV. Enforce diversity so the model handles varied inputs, aim for at least 10,000 pairs or ~10MB, and prioritise quality by removing PII, duplicates, and low-quality samples. Source public data from the Hugging Face Datasets hub for learning, and use proprietary data for production. Always reserve a validation split to test generalisation.
How do I provision GPU compute for fine-tuning?
Use RunPod first — it's affordable, reliable, and secure — then consider Vast.ai (cheapest but community-cloud security concerns), Lambda Labs, or AWS SageMaker. For a 7B model with QLoRA, a single A100 80GB (~$2/hr on RunPod) is sufficient. Spin up a JupyterLab instance from the provider dashboard, then install Axolotl and configure your run.
How do I monitor and validate a fine-tuning run?
During training, watch train_loss (should decrease), tokens per second, and steps per second — enable Weights & Biases in the YAML for tracking. After training, inspect the output directory for adapter_model.safetensors and adapter_config.json. Test the adapter with the built-in Gradio interface or Hugging Face PEFT, running prompts similar to but not identical to training data to confirm generalisation over memorisation.
// Troubleshooting
Why am I getting out-of-memory (OOM) errors during training?
OOM errors mean your effective batch size or model precision exceeds available VRAM. Reduce micro_batch_size to 1 and increase gradient_accumulation_steps to maintain the effective batch size. Enable QLoRA (load_in_4bit: true) if not already using it, and lower lora_r. These changes trade some speed or capacity for feasibility on smaller GPUs.
Why does my fine-tuning fail with flash-attention errors?
Flash-attention version 1.x causes errors in Axolotl. Always install flash-attention 2.0 or higher. Verify the installed version after running pip install -e '.[flash-attn,deepspeed]'. This is one of the most common setup pitfalls and is easy to miss because the error messages can be misleading.
Why can't Axolotl find my LoRA target modules?
Module naming conventions like q_proj and v_proj differ across model families, so a config that works for Llama may fail for another architecture. Always inspect the exact layer names with [name for name in model.state_dict().keys()] before setting lora_target_modules. Setting names that don't exist means LoRA silently adapts nothing or throws an error.
Why does my fine-tuned model underperform on new inputs?
Underperformance on unseen inputs usually stems from too low a LoRA rank, insufficient data quality or diversity, or too few epochs. Increase lora_r to 16 or 32 for complex datasets, expand and clean your dataset, and add training epochs. Also confirm your validation prompts genuinely represent the target task rather than edge cases the training data never covered.
Why is my training loss decreasing but the model still gives bad answers?
A falling training loss with poor outputs often signals memorisation rather than learning — the model fits training examples but can't generalise. Reduce epochs, add dropout or weight decay, and diversify the dataset. It can also mean your validation prompts are too different from the training distribution, so verify your dataset actually covers the tasks you're testing.
// Comparisons
How does LoRA compare to full fine-tuning?
Full fine-tuning updates every model weight, requiring hundreds of GBs of VRAM and risking catastrophic forgetting of base knowledge. LoRA freezes pre-trained weights and trains only small injected rank decomposition matrices, cutting trainable parameters by up to 10,000× and memory by ~3× while preserving base knowledge. For most use cases LoRA or QLoRA matches full fine-tuning quality at a fraction of the cost.
How does QLoRA compare to LoRA in terms of quality?
QLoRA loads the base model in 4-bit quantisation, so it introduces minor performance degradation compared to LoRA's full-precision base, but the difference is usually small and acceptable. QLoRA's advantage is dramatic memory savings — enabling 70B fine-tuning on 2× RTX 3090s. Choose LoRA when you have ample GPU memory and want maximum fidelity; choose QLoRA when compute is constrained.
How does fine-tuning a small high-quality model compare to a large one on noisy data?
The Phi/Orca principle shows that smaller models (1.5B-3B) fine-tuned on high-quality, diverse, well-structured data can outperform larger models trained on noisy data. Quality data reduces compute cost and speeds iteration. Prioritise clean, varied instruction-response pairs over raw dataset size — a curated 500-sample set can beat thousands of low-quality examples.
Should I use RunPod or Vast.ai for fine-tuning?
Use RunPod for most projects — it balances affordability, reliability, and security on its secure cloud. Vast.ai is cheaper but runs on a community cloud with security concerns, making it better for non-sensitive experiments than production data. Lambda Labs is reliable but has poor support, and AWS SageMaker suits enterprise workflows already on AWS.
// Advanced
How do I choose between targeting only attention modules versus all linear layers?
Targeting only q_proj and v_proj is the minimal, memory-efficient approach that works well for most instruction tuning. Add k_proj, o_proj, and the MLP modules (gate_proj, down_proj, up_proj) when the task is complex or the model underperforms. Only target embed_tokens and lm_head when your dataset uses custom vocabulary or syntax, since those layers are large and add significant compute.
How does double quantisation reduce memory further in QLoRA?
Double quantisation (bnb_4bit_use_double_quant: true) quantises the quantisation constants themselves, shaving additional bytes per parameter off the average memory footprint. Combined with nf4 quantisation, it lets you fit larger models on the same GPU with negligible extra quality loss. Enable it whenever you're memory-constrained and already using 4-bit loading.
How do I calculate the effective batch size and number of weight updates?
Effective batch size = micro_batch_size × gradient_accumulation_steps × num_GPUs. One epoch equals dataset_size divided by effective batch size, which is also the number of weight updates per epoch. Understanding this lets you tune memory versus training stability: a larger effective batch smooths gradients but needs more memory, which gradient accumulation compensates for.
When should I merge the LoRA adapter into the base model versus keep it separate?
Keep the adapter separate for portability — the rank decomposition matrices are tiny and easy to share, swap, or stack across contexts without carrying the full model. Merge the adapter into the base model when you want a single deployable artifact with no PEFT dependency at inference, or for slightly faster inference. Both are valid; portability favours keeping them separate.
How do I avoid catastrophic forgetting during fine-tuning?
Use LoRA or QLoRA, which freeze pre-trained weights and only train injected update matrices, so the model retains its base language knowledge. Keep lora_alpha moderate to avoid over-weighting new data, limit epochs, and maintain dataset diversity. Full fine-tuning with a high learning rate is the main cause of catastrophic forgetting — the LoRA approach is specifically designed to prevent it.