Aishwarya Srinivasan Watch original video

Aishwarya Srinivasan LLM Fine-Tuning Decision Framework

Last updated: 14 August 2026

Choose the right fine-tuning methodology for any LLM use case and understand exactly what is happening under the hood — so you are not just making API calls but engineering with genuine depth.

// TL;DR

The Aishwarya Srinivasan LLM Fine-Tuning Decision Framework is a decision tree for choosing the right fine-tuning method for any LLM use case. Use it before starting any fine-tuning project to avoid wasted compute and set up proper evaluation. It first checks whether your model is open-weight or closed-weight, forces a prompt/RAG baseline, then routes your task to one of three methods: QLoRA (PEFT) for domain/task adaptation, DPO for preference alignment, or Reinforcement Fine-Tuning with Verifiable Rewards for reasoning tasks with checkable answers. Data quality is the ceiling on every outcome.

Framework

// When should you use the LLM fine-tuning decision framework?

Use this skill whenever you need to adapt a pre-trained LLM to a specific task, domain, tone, or behaviour. Trigger it before starting any fine-tuning project to select the correct method, avoid wasted compute, and set up proper evaluation.

// What do you need before starting an LLM fine-tuning project?

Target task descriptionrequired
What the fine-tuned model needs to do — e.g. medical Q&A, code generation, customer-support tone
Model type (open-weight vs closed-weight)required
Whether you have access to model weights (Llama, Mistral, Qwen, DeepSeek, GLM, etc.) or are using a closed API (GPT, Claude, Gemini)
Available GPU infrastructurerequired
Number and type of GPUs available — e.g. single H100, multi-GPU cluster, or cloud API only
Performance goal and metricrequired
Specific, measurable target — accuracy, win-rate, safety score, tone consistency — needed for eval baseline
Data assetrequired
Training dataset available — its size, quality, and whether answers are verifiable or require human preference labels
Prompt + RAG baseline results
Results already achieved via prompt optimisation and context injection, to benchmark fine-tuning gains against

// What core principles drive fine-tuning method selection?

Pre-Training vs Post-Training Distinction

Pre-training is what foundational labs (OpenAI, Anthropic, Meta, Google) do on massive GPU clusters with trillions of tokens. What you receive — whether from Hugging Face or an API — is the pre-trained base model. Fine-tuning is post-training: you are adapting a model that already carries enormous knowledge, not building from zero.

Model-as-a-Service Reality

With LLMs the game has shifted from owning the entire training pipeline to adapting a foundational model served to you via API. You are not training from scratch anymore — you are steering existing knowledge toward your specific use case.

Parameter Scale Changes Everything

Traditional ML models had millions of parameters. LLMs run from a few billion (small open-source) to 100s of billions (frontier) to over a trillion (models like GLM). Updating even a fraction of a trillion-parameter model is not something you can casually do on a GPU — this is why methodology selection is critical.

Open-Weight vs Closed-Weight Constraint

Open-weight models (Llama, Mistral, Qwen, DeepSeek, KM, GLM) expose their weights, so you can download, fine-tune, and deploy them on your own infrastructure. Closed-weight models (GPT, Claude, Gemini) do not — you must use the provider's native fine-tuning service, submit your data, and get back a fine-tuned version via their API, with no visibility into what happens under the hood.

Prompt and Context First

Before you even start fine-tuning, optimise your prompts and then optimise your context (RAG). These should give you a decent performance bump and also act as a benchmark so you can measure exactly how much improvement fine-tuning actually adds.

Data Quality is the Ceiling

Fine-tuning with a bad dataset on a good model makes the model bad, not better. Data quality is the single biggest challenge across all fine-tuning methods — no technique compensates for poor training data.

Evaluation Before You Start

You need a proper eval set and a clear metric before you begin fine-tuning. Without them you will have no idea whether fine-tuning actually improved anything.

// How do you apply the LLM fine-tuning framework step by step?

1
Establish your prompt and RAG baseline
Run your task with optimised prompts and context injection first. Record your eval metric. This is your benchmark. Do not skip this — it tells you whether fine-tuning is even necessary and gives you a comparable number to beat.
2
Determine whether your model is open-weight or closed-weight
Open-weight models (Llama, Mistral, Qwen, DeepSeek, GLM, etc.) allow full fine-tuning on your own infrastructure. Closed-weight models (GPT, Claude, Gemini) require you to use the provider's native fine-tuning service — you submit data, they run the job, you get back a fine-tuned API endpoint with no under-the-hood visibility.
3
Define your eval set and success metric
Before touching a single weight, lock down what 'better' means. Choose a held-out eval set representative of your production task and a concrete metric (accuracy, win-rate, BLEU, safety pass-rate, etc.). Without this, fine-tuning is directionless.
4
Assess your task type to select the fine-tuning category
Ask three questions: (A) Is the goal domain adaptation or task-specific performance? → go to Step 5. (B) Do you care about aligning to human preferences — style, safety, tone? → go to Step 6. (C) Are you training a reasoning model on tasks with automatically verifiable answers (math, code)? → go to Step 7.
5
Apply Parameter-Efficient Fine-Tuning (PEFT) — start with QLoRA
For domain adaptation or task/tone improvement, start with QLoRA (Quantized Low-Rank Adaptation). It quantises the frozen base model to 4-bit precision and runs LoRA small trainable matrices on top. You can fine-tune a 70B-parameter model on a single H100 this way. Use Hugging Face's PEFT library. Only escalate to Full Fine-Tuning if QLoRA does not hit your quality bar — which is rare in practice.
6
Apply Preference Optimisation — start with DPO
When you need to align the model to human preferences (style, safety, tone), use DPO (Direct Preference Optimization) as your default. DPO skips training a separate reward model and directly optimises the language model on preference pairs in a single step — much simpler and less compute than RLHF. Use RLHF (with PPO) only if you have large volumes of high-quality human preference data and the engineering capacity to manage a complex training loop.
7
Apply Reinforcement Fine-Tuning with Verifiable Rewards
When your task has automatically verifiable answers (math problems, coding challenges with runnable tests), use reinforcement fine-tuning with verifiable rewards. The model generates multiple attempts; correct ones are rewarded, wrong ones are penalised. No human labellers needed. This is the technique behind OpenAI o-series and DeepSeek R1.
8
Audit your training data for quality before launching any job
Fine-tuning with a bad dataset on a good model makes the model bad, not better. Review for consistency, correctness, coverage of edge cases, and format alignment with your target behaviour. This is the biggest lever you have.
9
Run the fine-tuning job and evaluate against your Step 3 baseline
Compare your fine-tuned model against the prompt/RAG baseline on your held-out eval set. If gains are insufficient, diagnose: data quality first, then method choice, then hyperparameters. Do not add compute before fixing data.

// What do real LLM fine-tuning decisions look like in practice?

A team wants to deploy an open-source LLM that answers questions in a company-specific technical domain with consistent formatting and vocabulary.

The model is open-weight, so full infrastructure control is possible. The goal is domain adaptation and task-specific tone — not preference alignment or reasoning. Default to QLoRA via Hugging Face PEFT. Freeze the base model weights, inject small trainable LoRA matrices, train on the domain corpus. Evaluate against a prompt-engineered baseline on a held-out domain Q&A set. Only escalate to Full Fine-Tuning if QLoRA output quality is insufficient.

A product team is using a closed-weight frontier model and wants a version that is safer and more aligned to their brand voice.

Because the model is closed-weight, fine-tuning must go through the provider's native fine-tuning service (e.g. OpenAI fine-tuning API). Submit curated preference data. Internally the provider likely applies DPO or RLHF — you do not control this. Set a clear safety/tone eval metric before submitting. Compare returned fine-tuned API endpoint against your prompt-optimised baseline.

A research team wants to build a model that solves multi-step mathematical reasoning problems with higher accuracy.

Tasks have verifiable answers (correct/incorrect is deterministic). Use Reinforcement Fine-Tuning with Verifiable Rewards. Let the model generate multiple solution attempts per problem; automatically score correctness; reward correct reasoning chains and penalise wrong ones. No human preference labellers required. This mirrors the methodology behind modern reasoning models.

A startup wants a chatbot that responds in a warmer, more empathetic tone and avoids certain unsafe outputs — they have human raters who can compare response pairs.

This is a preference alignment task. With human rater comparison data available, apply DPO (Direct Preference Optimization) as the default: create chosen/rejected response pairs, run DPO directly on the language model. Avoid full RLHF unless you have very large preference datasets and dedicated ML engineering capacity to manage the reward model + PPO training loop complexity.

// What mistakes should you avoid when fine-tuning an LLM?

Skipping prompt and RAG optimisation before fine-tuning — you lose your performance benchmark and may fine-tune unnecessarily.
Attempting to fine-tune a closed-weight model from your own infrastructure — you do not have access to the weights; you must use the provider's native fine-tuning service.
Fine-tuning with a bad dataset on a good model — this makes the model bad, not better. Data quality is the ceiling of your outcome.
Starting a fine-tuning job without a proper eval set and a clear metric — you will have no idea whether fine-tuning actually improved anything.
Defaulting to Full Fine-Tuning when QLoRA is sufficient — Full Fine-Tuning is expensive, slow, and requires serious GPU infrastructure; it is rarely necessary in practice.
Confusing RLHF and DPO — RLHF requires training a separate reward model and running PPO, making it complex and expensive; DPO skips the reward model and is the default choice for preference tuning on open-source models.
Applying Reinforcement Fine-Tuning with Verifiable Rewards to tasks that do not have automatically verifiable answers — this method only works when correctness can be checked without human labellers.
Assuming fine-tuning teaches a model new factual knowledge — fine-tuning adapts behaviour, style, and task performance; for knowledge injection, retrieval-augmented generation (RAG) is typically more appropriate.

// What key fine-tuning terms should you know?

Pre-Training: The massive, compute-intensive training run performed by foundational labs (OpenAI, Anthropic, Meta, Google) on trillions of tokens across huge GPU clusters. Produces the base model. You did not do this — the foundational lab did.
Post-Training: Everything that happens after pre-training. Fine-tuning is the most common form of post-training.
Base Model: The model produced by pre-training. What you receive out of the box from Hugging Face or an API.
Open-Weight Model: A model whose weights are publicly accessible (e.g. Llama, Mistral, Qwen, DeepSeek, KM, GLM). You can download, fine-tune, and deploy these on your own infrastructure.
Closed-Weight Model: A model whose weights are not publicly accessible (e.g. GPT series, Claude, Gemini). You can only fine-tune these through the provider's native fine-tuning service; you do not get visibility into the process.
Model-as-a-Service: The LLM paradigm where the pre-trained model is created by a foundational lab and served to you via an API. You adapt it rather than training from zero.
Parameter-Efficient Fine-Tuning (PEFT): A family of techniques that update only a small fraction of a model's parameters rather than all of them, drastically reducing compute, memory, and cost. LoRA and QLoRA are the primary examples.
LoRA (Low-Rank Adaptation): A PEFT technique that freezes the original model weights and injects small trainable matrices into specific layers. Only those small matrices are trained. At inference time they combine with the frozen weights to produce adapted behaviour.
QLoRA (Quantized Low-Rank Adaptation): LoRA with the frozen base model quantised down to 4-bit precision, further reducing memory requirements. Enables fine-tuning a 70B-parameter model on a single H100. The default starting point for most teams fine-tuning open-source models today.
Full Fine-Tuning: Updating every single parameter in the model. Provides maximum flexibility and usually the best quality, but is expensive, slow, and requires serious GPU infrastructure. Reach for this only when PEFT is not hitting your quality bar — which is rare in practice.
Reinforcement Fine-Tuning with Verifiable Rewards: A technique where the model generates multiple attempts at a task with automatically verifiable answers (math, code). Correct attempts are rewarded, wrong ones penalised. No human labellers needed. The technique behind OpenAI o-series and DeepSeek R1.
RLHF (Reinforcement Learning from Human Feedback): The classic post-training technique that powers ChatGPT. Human raters compare model outputs; a reward model is trained on that preference data; the reward model then guides the language model via PPO. Powerful but expensive and complex.
PPO (Proximal Policy Optimization): The core reinforcement learning algorithm used inside RLHF to update the language model based on reward model signals.
DPO (Direct Preference Optimization): A simpler alternative to RLHF that skips the reward model entirely. Directly optimises the language model on chosen/rejected response pairs in a single step. Less compute, easier to implement, and the default choice for preference tuning on open-source models.
Preference Optimisation: The category of fine-tuning techniques (DPO, RLHF) focused on aligning a model to human preferences — style, safety, tone — using comparison data rather than scalar labels.
Verifiable Rewards: Reward signals that can be computed automatically without human raters, because the task has a ground-truth answer that can be checked programmatically (e.g. running test cases on generated code).

// FREQUENTLY ASKED QUESTIONS

What is the LLM fine-tuning decision framework?

It's a structured decision tree for selecting the correct fine-tuning method for any LLM use case. It routes you based on your model type (open vs closed weight) and task type: QLoRA for domain and task adaptation, DPO for preference alignment, and Reinforcement Fine-Tuning with Verifiable Rewards for reasoning tasks. It forces a prompt/RAG baseline and an eval set before you touch any weights.

What is the difference between QLoRA, DPO, and RLHF?

QLoRA is parameter-efficient fine-tuning for domain and task adaptation — it freezes a 4-bit quantized base model and trains small LoRA matrices. DPO aligns a model to human preferences using chosen/rejected pairs without a reward model. RLHF also aligns to preferences but trains a separate reward model and runs PPO, making it far more complex and expensive than DPO.

How do I choose a fine-tuning method for my LLM?

First establish a prompt and RAG baseline, then confirm your model is open-weight or closed-weight, then define your eval set and metric. Next, classify your task: domain or task adaptation goes to QLoRA, human preference alignment goes to DPO, and reasoning tasks with verifiable answers go to Reinforcement Fine-Tuning with Verifiable Rewards. Audit your data quality before launching any job.

How do I fine-tune a closed-weight model like GPT or Claude?

You must use the provider's native fine-tuning service — you cannot fine-tune closed-weight models on your own infrastructure because you don't have the weights. Submit curated data to the provider, they run the job internally (likely DPO or RLHF), and you get back a fine-tuned API endpoint with no visibility into the process. Define your eval metric before submitting.

When should I fine-tune an LLM instead of using prompts or RAG?

Fine-tune only after optimizing prompts and context (RAG) first, since those give you a benchmark to beat. Use fine-tuning when you need consistent tone, domain-specific behavior, preference alignment, or reasoning improvements that prompting can't reliably achieve. For injecting new factual knowledge, use RAG instead — fine-tuning adapts behavior and style, not facts.

How does fine-tuning compare to RAG for adding knowledge?

RAG is usually better for adding new factual knowledge because fine-tuning adapts behavior, style, and task performance rather than reliably teaching facts. Fine-tuning changes how a model responds; RAG changes what information it can access at inference time. Use RAG for knowledge injection and fine-tuning for tone, format, domain adaptation, or preference alignment.

What results can I expect from fine-tuning an LLM?

You can expect improved task-specific performance, consistent tone and formatting, better preference alignment, or higher reasoning accuracy — but only measurable against your prompt/RAG baseline. Gains depend almost entirely on data quality; a bad dataset makes a good model worse. QLoRA can fine-tune a 70B model on a single H100, so many teams see strong gains without heavy infrastructure.

Can I fine-tune a 70B parameter model without a GPU cluster?

Yes — QLoRA (Quantized Low-Rank Adaptation) lets you fine-tune a 70B-parameter model on a single H100. It quantizes the frozen base model to 4-bit precision and trains only small LoRA matrices on top, drastically reducing memory. Use Hugging Face's PEFT library. Only escalate to Full Fine-Tuning if QLoRA fails to hit your quality bar, which is rare.

Why does data quality matter so much in fine-tuning?

Because data quality is the ceiling of your outcome — fine-tuning with a bad dataset on a good model makes the model bad, not better. No technique compensates for poor training data. Before launching any job, audit for consistency, correctness, edge-case coverage, and format alignment with your target behavior. This is the single biggest lever you have.

What is the difference between pre-training and fine-tuning?

Pre-training is the massive compute run foundational labs perform on trillions of tokens to produce a base model — you did not do this. Fine-tuning is post-training: you adapt a pre-trained model that already carries enormous knowledge toward your specific task. You're steering existing knowledge, not building from zero.

// GET THIS SKILL — FREE