AI Anytime LLM Fine-Tuning End-to-End Framework

// When should you use the LLM fine-tuning framework?

Use this skill whenever a user wants to fine-tune a large language model on a custom or domain-specific dataset and needs to select the right technique (full fine-tuning vs LoRA vs QLoRA), configure hyperparameters, choose compute infrastructure, and run training end-to-end. Also applies when a user is debugging an existing fine-tuning config or trying to reduce GPU memory requirements.

// What do you need before you start fine-tuning an LLM?

• Target LLMrequired
  The base model to fine-tune, e.g. Llama 2 7B, Mistral 7B, OpenLlama 3B. Must be a commercially usable model if building a product.
• Task-Specific Datasetrequired
  Labeled examples relevant to the desired task, typically instruction-response pairs. Minimum recommended size: 10,000 question-answer pairs or ~10 MB file size for a CSV.
• Fine-Tuning Techniquerequired
  Choice of full fine-tuning, LoRA, or QLoRA. Determined by available GPU memory and dataset complexity.
• Compute / GPU Setuprequired
  Cloud provider and GPU spec chosen to meet memory requirements (~150–195 GB for 7B models). Options: RunPod, Vast.ai, Lambda Labs, AWS SageMaker, Google Colab.
• Hyperparameter Config (YAML)
  LoRA rank (r), LoRA alpha, target modules, batch size, gradient accumulation steps, number of epochs, learning rate. Can be auto-populated from an Axolotl YAML template.

// What core principles drive effective LLM fine-tuning?

// How do you fine-tune an LLM step by step?

1. 1

   Define the fine-tuning objective and select the base model

   Choose a commercially usable open-source model (Llama 2 7B, Mistral 7B, OpenLlama 3B) for product use cases. Identify whether the task requires instruction-tuning (instruction-response pairs), domain adaptation, or code generation. Confirm the model license permits your use case.

2. 2

   Prepare and validate the task-specific dataset

   Gather labeled examples as instruction-response pairs (JSONL or CSV). Enforce: (a) diversity — multiple scenarios so the model handles varied inputs; (b) minimum size — 10,000 pairs or ~10 MB CSV; (c) quality — follow Phi/Orca principle: fewer high-quality samples beat many low-quality ones. Remove PII and duplicates. Source: Hugging Face Datasets hub for learning; proprietary data for production.

3. 3

   Select the fine-tuning technique based on compute budget

   Full fine-tuning: ~150–195 GB VRAM for 7B models — rarely feasible without enterprise hardware. LoRA: ~3× memory reduction vs full fine-tuning; use when you have multiple A100s. QLoRA (quantised LoRA): load model in 4-bit via bits-and-bytes; enables 7B–70B fine-tuning on consumer GPUs (2× RTX 3090 or 1× A100). Default recommendation for most users: QLoRA.

4. 4

   Provision GPU compute on a cloud provider

   Preferred order: RunPod (recommended — affordable, reliable, secure cloud option) → Vast.ai (cheapest but security concerns on community cloud) → Lambda Labs → AWS SageMaker. For 7B models: 1× A100 80 GB (~$2/hr on RunPod) is sufficient for QLoRA. A 2,000–3,000 row dataset trains in ~$5. Spin up a JupyterLab instance via the provider's dashboard.

5. 5

   Set up Axolotl as the low-code fine-tuning tool

   Clone the Axolotl GitHub repository into the JupyterLab environment. Install dependencies via: pip install packaging, then pip install -e '.[flash-attn,deepspeed]'. Ensure flash-attention >= 2.0 (version 1.x causes errors). Axolotl accepts a single YAML config file as input — this is the control surface for the entire fine-tuning run.

6. 6

   Configure the YAML file with model, dataset, and LoRA/QLoRA hyperparameters

   Navigate to examples/<your-model>/lora.yml or qlora.yml in the Axolotl repo. Set: base_model path or Hugging Face ID; dataset path (local or HF hub); load_in_4bit: true for QLoRA; LoRA hyperparameters (see step 7); num_epochs (start with 1 for testing); micro_batch_size (start with 1); gradient_accumulation_steps (start with 8); learning_rate (~2e-4 typical). Point dataset field to your JSONL/CSV file path or Hugging Face dataset name.

7. 7

   Set LoRA hyperparameters deliberately

   lora_r (LoRA rank): start at 8 (paper default); increase to 16 or 32 for complex datasets; higher rank = better results but more compute. lora_alpha (scaling factor): typically 2× lora_r; lower value preserves more base model knowledge; higher value shifts toward new data. lora_target_modules: at minimum target q_proj (query projection) and v_proj (value projection); optionally add k_proj, o_proj, gate_proj, down_proj, up_proj. If dataset has custom syntax, also target embed_tokens and lm_head. Find exact module names via: [name for name in model.state_dict().keys()].

8. 8

   Configure QLoRA-specific settings if using quantisation

   Set load_in_4bit: true. Set bnb_4bit_quant_type: nf4 (4-bit NormalFloat — optimally handles normally distributed weights). Enable double quantisation (bnb_4bit_use_double_quant: true) to further reduce average memory footprint by quantising the quantisation constants. bits-and-bytes library handles on-the-fly near-lossless quantisation. Expect minor performance degradation — treat as acceptable tradeoff for feasibility.

9. 9

   Set batch size and gradient accumulation to manage memory

   Effective batch size = micro_batch_size × gradient_accumulation_steps × num_GPUs. If GPU OOM errors occur: reduce micro_batch_size to 1, increase gradient_accumulation_steps to 8 or 16 to maintain effective batch size. Understand: one epoch = (dataset_size / batch_size) batches = that many weight updates. Larger batch size → higher GPU memory → use gradient accumulation to compensate.

10. 10

    Launch fine-tuning and monitor training

    Run: accelerate launch -m axolotl.cli.train examples/<model>/qlora.yml. Monitor: train_loss (should decrease), tokens per second, steps per second. Ignore standard warnings. Training a 7B model for 1 epoch on ~2,000–3,000 rows takes approximately 1–2 hours on 1× A100. Checkpoints are saved to the output directory (e.g. qlora-out/). Enable Weights & Biases (wandb) in the YAML for experiment tracking.

11. 11

    Inspect outputs and validate the fine-tuned adapter

    Output directory contains: adapter_model.bin or .safetensors (the LoRA weights), adapter_config.json, optimizer.pt, scheduler.pt. These LoRA adapter weights are portable — they can be merged with the base model or used standalone. Test via the built-in Gradio interface (axolotl.cli.inference) or load the adapter with Hugging Face PEFT library. Run sample prompts similar to — but not identical to — training data to verify generalisation, not memorisation.

12. 12

    Iterate on hyperparameters based on results

    If model memorises training data: increase dataset diversity, add dropout/weight decay, reduce epochs. If model underperforms on new inputs: increase lora_r, increase dataset size/quality, increase epochs. If OOM errors: reduce lora_r, reduce micro_batch_size, enable QLoRA if not already using it. Learning rate too small → slow convergence or local minima. Learning rate too large → unstable training.

// What do real LLM fine-tuning projects look like?

// What mistakes should you avoid when fine-tuning an LLM?

• Treating fine-tuning as cheap: full fine-tuning of multi-billion parameter models requires hundreds of GBs of VRAM — do not attempt without LoRA or QLoRA unless you have enterprise compute.
• Setting LoRA rank (lora_r) too low for a complex dataset: rank=8 is the paper minimum; complex datasets with rich relational structure need rank=16 or 32.
• Confusing batch size with epoch: batch size is samples processed per update; epoch is one full pass through the entire dataset. These are independent hyperparameters.
• Using flash-attention 1.x instead of 2.x: flash-attention version 1 causes errors in Axolotl — always install flash-attention >= 2.0.
• Ignoring data diversity: training a chat model on a single type of conversation will cause failure on unseen input types — the model cannot generalise what it has not seen.
• Memorisation instead of generalisation: too many epochs on a small dataset causes the model to memorise training examples rather than learn patterns — use early stopping and validation splits.
• Not checking target module names per model: q_proj, v_proj naming conventions differ across model families — always inspect model.state_dict().keys() to confirm exact layer names before setting lora_target_modules.
• Skipping the fundamentals: using low-code tools like Axolotl without understanding pre-training, tokenisation, gradient descent, and LoRA mechanics leads to inability to debug errors or tune performance.
• Choosing Lambda Labs for ease: Lambda Labs has poor support — RunPod is the recommended provider for reliability and affordability.
• Assuming QLoRA has zero performance degradation: quantisation introduces minor performance reduction; treat it as an acceptable tradeoff, but do not assume lossless equivalence to full precision training.

// What are the key LLM fine-tuning terms you should know?

Pre-Training

The initial training phase where a model learns general language knowledge (next-word prediction or masked token prediction) on terabytes of text data. Produces a model with general language understanding but no task-specific knowledge.

Fine-Tuning

The downstream training phase that specialises a pre-trained model's capabilities on a task-specific, labeled dataset (e.g. instruction-response pairs) by adjusting model weights using gradient-based optimisation. Data size is significantly smaller than pre-training.

LoRA (Low-Rank Adaptation)

A training method that reduces memory consumption by injecting pairs of rank decomposition matrices (update matrices) into the model's existing layers, rather than updating all pre-trained weights. Reduces trainable parameters by up to 10,000× and GPU memory by ~3× vs. full fine-tuning.

QLoRA (Quantised Low-Rank Adaptation)

LoRA applied to a model loaded in 4-bit quantisation via the bits-and-bytes library. Achieves near-lossless quantisation and enables fine-tuning of 70B models on 2× RTX 3090s instead of 16× A100s.

LoRA Rank (lora_r)

Hyperparameter that determines the number of rank decomposition matrices applied to weight matrices. Higher rank = better results but more compute. Paper recommends r=8 as default; complex datasets may require r=16 or r=32.

LoRA Alpha (lora_alpha)

Scaling factor that adjusts the contribution of the update matrices during training. Lower value gives more weight to original pre-trained data (preserves base knowledge); higher value shifts the model more toward new training data.

LoRA Target Modules

The specific weight matrices selected for LoRA adaptation. Minimum: q_proj (query projection) and v_proj (value projection). Extended targets include k_proj, o_proj, gate_proj, down_proj, up_proj, embed_tokens, lm_head, and norm layers depending on dataset complexity.

q_proj (Query Projection)

The projection matrix applied to query vectors in the Transformer attention mechanism. Transforms input hidden states to the desired dimensions for query representation. One of the most important LoRA target modules.

v_proj (Value Projection)

The projection matrix applied to value vectors in the Transformer attention mechanism. Transforms input hidden states for effective value representation. Paired with q_proj as the minimum LoRA target.

lm_head

The output layer of a language model responsible for generating predictions/scores for the next token. Critical to target in LoRA if the dataset contains custom syntax or specialised output formats.

embed_tokens

Parameters in the embedding layer that map input tokens to their vector representations. Important LoRA target when the dataset uses custom vocabulary or syntax.

nf4 (4-bit NormalFloat)

A quantisation data type used in QLoRA that optimally handles normally distributed model weights. Enabled via bits-and-bytes library for near-lossless 4-bit quantisation.

Instruction Tuning (Instruction-Tuned Model)

A fine-tuning approach where the training dataset consists of instruction-response pairs, teaching the model to follow natural language instructions. The dominant paradigm for chat and assistant models.

Gradient Accumulation Steps

A mechanism to simulate large batch sizes without high VRAM: the global batch is split into sequential mini-batches, gradients are accumulated across them, and a single weight update is performed. Effective batch size = micro_batch_size × gradient_accumulation_steps × num_GPUs.

Causal Language Modeling (CLM)

The pre-training objective focused on predicting the next word in a sequence given the preceding context. Underpins autoregressive generation models. Distinct from Masked Language Modeling (MLM), which predicts masked tokens.

Masked Language Modeling (MLM)

A pre-training objective where certain tokens are intentionally hidden and the model must predict them from surrounding context. Used in encoder-style models; contrasted with Causal Language Modeling.

Catastrophic Forgetting

The risk that fine-tuning on new data causes the model to lose its previously learned general language knowledge. LoRA mitigates this by keeping pre-trained weights frozen and only training the injected update matrices.

Axolotl

A low-code fine-tuning tool that accepts a single YAML configuration file as input and handles the full fine-tuning pipeline (data loading, tokenisation, LoRA/QLoRA setup, training, checkpointing). The recommended tool in this framework for productivity and efficiency.

bits-and-bytes (bnb)

A Python library used by QLoRA to perform on-the-fly 4-bit quantisation of language models, enabling near-lossless model compression and fine-tuning on consumer-grade GPUs.

Rank Decomposition Matrices (Update Matrices)

The pairs of low-rank matrices injected by LoRA into existing model layers. These are the only weights trained during LoRA fine-tuning; the original model weights remain frozen.

Portability of Trained Weights

A LoRA advantage: the rank decomposition matrices have far fewer parameters than the original model, making them easy to transfer, share, and use across different contexts without carrying the full model.

// FREQUENTLY ASKED QUESTIONS