Hugging Face Agent-Driven SFT Training Framework

Last updated: 11 July 2026

Apply a repeatable, agent-orchestrated Supervised Fine-Tuning pipeline to train a small open model on agentic traces — without writing training scripts by hand — and produce a focused, deployable model adapter with tracked metrics and eval scores.

// TL;DR

The Hugging Face Agent-Driven SFT Training Framework is a repeatable pipeline for fine-tuning a small open LLM to imitate agentic behaviour — tool calls, multi-turn reasoning, harness-specific formatting — using collected agent traces. Instead of writing training scripts by hand, you give a coding agent a Contract Prompt specifying the base model, trace dataset, hyperparameter sweep, tracking destination, and eval criteria. The agent plans, writes, runs, and evaluates while you verify artifacts. Use it when you have a specific domain task and want to focus a general model on that use case without full RL training, producing a deployable adapter with tracked metrics and eval scores.

Framework

// When should you use the agent-driven SFT training framework?

Use this skill whenever you want to fine-tune a small LLM to imitate agentic behaviour (tool calls, multi-turn reasoning, harness-specific formatting) from a collected set of real or synthetic agent traces. Especially relevant when you have a specific domain or task and want to focus a general model on that use case without full RL training.

// What inputs do you need to run an agent-driven SFT training run?

base_model_idrequired
Hugging Face model ID of the small instruction-tuned model to fine-tune (e.g. google/gemma-4-2b-it)
trace_dataset_idrequired
Hugging Face dataset ID containing agentic traces in multi-turn conversation format (prompt + tool calls + completions)
trackio_project_namerequired
Name for the Tracchio (track.io) project where all run metrics will be grouped
hf_adapter_repo_prefixrequired
HF Hub repository prefix where each trained adapter will be pushed
parameter_sweep_rangesrequired
Ranges or candidate values for learning rate, LoRA rank, and sequence length to sweep over
eval_benchmarks
Benchmark(s) to run on final weights (e.g. HumanEval, MBPP) plus any domain-specific held-out eval
agent_skills_repo
Local or remote repo containing skill files that instruct the agent how to use TRL, HF Jobs, and Tracchio correctly
budget_and_hardware_constraints
GPU tier preference and spend ceiling to pass to the agent as constraints

// What core principles guide agent-driven SFT training?

One Abstraction Level Up

Instead of writing Python training scripts step-by-step, give a coding agent a high-level prompt that defines the desired outcome and constraints. The agent plans, writes, runs, and evaluates — you operate as a human operator verifying artifacts, not a script author.

The Contract Prompt

The agent prompt is a formal contract: it specifies the model, dataset, hardware, tracking destination, adapter push targets, eval criteria, and a rule that the best run is selected by held-out eval loss — not by the agent's intuition. Every constraint the agent must honour should be stated explicitly in this contract.

Orient Before You Spend

Before launching any paid HF Jobs, the agent must verify: the correct HF model ID resolves, the dataset exists and has the expected shape, push/job authorizations are confirmed, and a smoke-test run passes without OOM errors. Never let the agent skip this orientation phase.

SFT as Emulation, Not Creation

SFT teaches a model to imitate the completions in the trace dataset — it cannot produce outputs better than the traces it learns from. The training objective is to produce the best imitator of those traces, not the best agent. The proxy metric is held-out loss; a lower loss means closer imitation.

Masking Is the Defining Act of SFT

The core mechanic that separates SFT from pre-training is masking the user-side tokens (prompt) with -100 labels so the loss is only calculated on the assistant completions. The model learns to generate the model's responses, not the user's instructions.

Skills as Agent Memory

Skill files in the agent's working repo encode tested, efficient workflows (how to use TRL, HF Jobs, Tracchio). They guide the agent away from common failure routes. Update skills continuously — especially to record mistakes the agent frequently makes — so those routes are avoided in future runs.

Sequence Length Is a First-Class Hyperparameter for Agentic Traces

Agentic traces are long sequences. Choices around context window length and truncation strategy materially affect what the model learns. Always include sequence length in the parameter sweep alongside learning rate and LoRA rank.

Mixed Evals Protect General Capability

When fine-tuning for a specific domain, use a domain-specific benchmark to track improvement AND at least one general benchmark (e.g. coding or terminal usage) to confirm the model's general abilities have not been compromised.

Clean Traces Before Publishing

Real agent traces may contain secrets, API tokens, or credentials. Before pushing any trace dataset to a public HF Hub repo, run a scrubbing tool (e.g. Presidio HF) to detect and redact sensitive values.

// How do you run agent-driven SFT step by step?

1
Collect or source agentic traces
Traces must be in multi-turn conversation format: user instruction → assistant tool calls + responses → continuation. They can come from your own agent sessions (any harness, any model) or a public dataset of real sessions. Ensure traces are scrubbed of secrets using a tool like Presidio HF before pushing to a public dataset. Push the cleaned dataset to HF Hub.
2
Prepare the agent's skills repo
Create or clone a working directory containing skill files for: (a) SFT workflow — training stages, data format, masking approach; (b) TRL usage — SFTConfig and SFTTrainer patterns; (c) HF Jobs — how to schedule, smoke-test, and retrieve results; (d) Tracchio observability — how to log metrics consistently. Install skills via TRL CLI or HF CLI where possible. Add a skill entry for any recurring agent mistake you want to prevent.
3
Write the Contract Prompt
The Contract Prompt must specify: base model HF ID, dataset HF ID, SFT algorithm, parameter sweep ranges (learning rate, LoRA rank, sequence length), HF Jobs hardware tier and budget ceiling, Tracchio project name (all runs must share one project), adapter push destination per run, final model selection rule (best held-out eval loss), eval benchmarks to run on final weights, and a requirement to add eval scores + a summary table (job IDs, Tracchio links, repo links, scores) to the model card README. State that the agent must orient itself before spending.
4
Trigger the agent and monitor orientation phase
Send the Contract Prompt to your coding agent (e.g. Codex, Claude, or any tool-use capable agent) pointed at the skills repo. Verify the agent completes orientation before any jobs run: it should resolve the exact HF model ID, confirm dataset exists and shape is correct, confirm push and HF Jobs authorizations, and execute at least one smoke-test job to check OOM and formatting. If the agent skips orientation, intervene and add an explicit orientation rule to the Contract Prompt.
5
Review the agent-generated training script
The agent will produce a TRL-based script using SFTConfig and SFTTrainer. Verify it: applies the chat template via apply_chat_template, masks user-side tokens with -100 labels, logs to the correct Tracchio project, and pushes each adapter to the specified HF repo. You do not need to write this script — but you must be able to read and validate it.
6
Run parameter sweep via HF Jobs
The agent schedules multiple HF Jobs, one per parameter combination. Each job trains the model, logs metrics live to Tracchio, and pushes its adapter. Monitor the Tracchio dashboard during runs: loss should decrease, token accuracy should increase, entropy should decrease. All runs must appear in the same Tracchio project for comparison. If any run errors, the agent should diagnose and retry — verify it does so rather than silently dropping failed runs.
7
Inspect Tracchio metrics and validate training health
In the Tracchio dashboard check: (a) Train loss decreasing — confirms the model is learning completions; (b) Eval loss tracking similarly to train loss — confirms no significant overfitting; (c) Token accuracy increasing; (d) Entropy decreasing; (e) Learning rate schedule behaving as expected (high early, tapering). If loss barely moves, consider more steps, higher learning rate, or longer sequence length. If train and eval loss diverge, the model is overfitting — reduce steps or increase data diversity.
8
Select best adapter by held-out eval loss
The agent selects the best run using held-out loss as the proxy metric — the best imitator of the traces. This is defined in the Contract Prompt. Do not override this selection manually unless you have a specific reason. Push the final weights (not just adapter) of the winning run to HF Hub.
9
Run evals on final weights
Trigger eval jobs via HF Jobs using Inspect AI (or equivalent). Run at minimum: one domain-relevant benchmark (or held-out task set) and one general capability benchmark (e.g. HumanEval, MBPP for coding). The agent should record scores and add them to the model card README in a summary table with: job IDs, Tracchio dashboard link, final adapter repo link, eval scores per benchmark.
10
Record lessons and update skills
After the run, review the full agent trace (save it as an HF bucket or dataset for later study). Identify any routes the agent went down that wasted compute or caused errors. Add entries to the relevant skill files to prevent repetition. This is how the skills repo compounds over time — it is living documentation of your best-known SFT workflow.

// What are real examples of agent-driven SFT in action?

A developer wants to fine-tune a 2B parameter open model to handle pull-request triage in a specific codebase, using real traces from their own agent sessions doing triage tasks.

Scrub the collected triage traces of any repo secrets, push the dataset to HF Hub. Write a Contract Prompt specifying the 2B model, the triage dataset, a sweep over learning rate [1e-4, 5e-5] and LoRA rank [8, 16], sequence length sized to cover full PR diffs, one domain benchmark (triage accuracy on held-out PRs) and one general benchmark (coding task pass-rate) to protect general ability. Point a coding agent at a skills repo containing TRL and HF Jobs skills, trigger it, monitor orientation phase, then watch Tracchio for loss and token accuracy. Select the best adapter by held-out loss, run evals, publish the model card with scores.

A researcher wants to train a small model to follow a specific agent harness's tool-call format using publicly available traces from that harness.

Locate the public trace dataset on HF Hub (verify it is scrubbed). Write a Contract Prompt that specifies the target harness's chat template format as a constraint in the SFT workflow skill, sweeps sequence length aggressively (agentic traces are long), and uses a harness-specific eval (can the fine-tuned model complete tasks in that harness?) as the primary benchmark. The agent writes a script that applies the harness's chat template via apply_chat_template and masks all user-turn tokens. Monitor entropy decrease in Tracchio as evidence the model is converging on the harness's tool-call language.

// What mistakes should you avoid when fine-tuning agents with SFT?

Skipping the orientation phase and letting the agent launch paid HF Jobs before confirming model ID, dataset shape, and authorizations — this wastes budget on runs that will fail.
Not masking user-side tokens with -100 labels — the model will learn to generate the user's instructions instead of the assistant's completions, producing a broken fine-tune.
Treating the held-out loss as a measure of agent quality rather than imitation quality — SFT produces the best imitator of the traces, not the best agent; do not over-interpret low loss as proof of a capable agent.
Ignoring sequence length as a hyperparameter — agentic traces are long; truncation choices directly affect what the model learns and must be included in the sweep.
Pushing trace datasets containing API tokens, secrets, or credentials to public HF Hub repos without scrubbing first.
Using only a domain benchmark for evals and omitting a general capability benchmark — the model may improve on the target task while losing general coding or reasoning ability undetected.
Allowing the agent to proceed without skills files — without skills, the agent will make avoidable mistakes (wrong TRL config patterns, inconsistent Tracchio logging) that are expensive to diagnose after the fact.
Expecting SFT alone to produce a capable autonomous agent — SFT is step one (imitation); RL and environment-based training are required for genuine agent capability improvement.
Running too few steps or too narrow a learning rate sweep and concluding SFT doesn't work — check Tracchio for whether loss was still decreasing at the end of the run before drawing conclusions.
Not saving the agent trace from the orchestration run itself — the full trace of the coding agent's decisions is a valuable artifact for debugging and for building future skills.

// What key terms should you know for agent-driven SFT?

SFT (Supervised Fine-Tuning): A continuation of pre-training where the model learns to predict assistant completions from prompt-completion pairs. Unlike pre-training on arbitrary text, SFT uses structured instruction-response traces and masks user-side tokens so the model only learns to generate the assistant's side.
Agentic Traces: Saved recordings of real agent sessions: multi-turn conversations containing user instructions, assistant tool calls, tool results, and continuations. The raw material for SFT training in this framework.
Contract Prompt: The high-level agent prompt that defines all constraints the agent must honour during a training run: model, dataset, hardware, sweep ranges, tracking destination, push targets, selection criteria, and eval requirements. Acts as the formal specification between the human operator and the agent.
Orientation Phase: The mandatory pre-spend verification stage where the agent confirms: correct model ID, dataset existence and shape, push/job authorizations, and a smoke-test job succeeds — before any real training jobs are launched.
Smoke Test: A minimal, cheap HF Job run executed during orientation to confirm infrastructure, formatting, and authorization work correctly before full parameter sweep jobs are launched.
Parameter Sweep: A set of HF Jobs run in parallel, each with a different combination of hyperparameters (learning rate, LoRA rank, sequence length), all tracked in the same Tracchio project so results can be compared.
Skills (Skill Files): Markdown or text files in the agent's working repo that encode tested workflows and guardrails — how to use TRL, HF Jobs, Tracchio, etc. They shape the agent's decisions and are updated over time to record and prevent recurring mistakes.
Best Imitator: The selection criterion for the winning adapter: the run with the lowest held-out eval loss, meaning it most closely reproduces the completions in the trace dataset. SFT optimises for imitation, not for agent task performance.
Masking / Labels (-100): The core SFT mechanic: user-side tokens in the training sequence are replaced with -100 in the labels tensor, instructing the loss function to ignore them. Only assistant-side tokens contribute to the cross-entropy loss and weight updates.
Tracchio (track.io): A lightweight, free experiment tracking tool used to log live metrics from each HF Jobs training run. All runs in a sweep must share one Tracchio project for comparison. Can be self-hosted on Hugging Face Spaces.
HF Jobs: Hugging Face's remote compute job scheduling service, used to run training and eval scripts on GPU hardware without requiring local compute. The agent uses the HF CLI to schedule, monitor, and retrieve results from jobs.
LoRA / PEFT: Parameter-Efficient Fine-Tuning via Low-Rank Adaptation — attaches small trainable adapter matrices to the frozen base model. LoRA rank is a key hyperparameter in the sweep; only adapters are pushed per run until the best is selected for full-weight push.
apply_chat_template: A Hugging Face Transformers tokenizer method that formats a multi-turn conversation into the model's expected chat format, producing the encoded input with correct special tokens before masking and training.
Held-out Eval Loss: The cross-entropy loss computed on a portion of the trace dataset withheld from training. Used as the proxy metric to select the best adapter — lower means closer imitation of the traces.
Inspect AI: The evaluation framework used in this workflow to run benchmarks (HumanEval, MBPP) on final model weights via HF Jobs, producing the scores that are recorded in the model card.

// FREQUENTLY ASKED QUESTIONS

What is the Hugging Face Agent-Driven SFT Training Framework?

It's a repeatable pipeline where a coding agent orchestrates Supervised Fine-Tuning of a small open LLM on agentic traces — instead of you writing training scripts by hand. You provide a Contract Prompt specifying the model, dataset, hyperparameter sweep, tracking, and eval rules. The agent plans, runs, and evaluates, producing a deployable model adapter with tracked metrics and benchmark scores while you verify artifacts.

What are agentic traces in SFT training?

Agentic traces are saved recordings of real agent sessions: multi-turn conversations containing user instructions, assistant tool calls, tool results, and continuations. They are the raw training material in this framework. The model learns to imitate the assistant-side completions in these traces, teaching it tool-call formatting and multi-turn reasoning specific to your domain or agent harness.

How do I fine-tune a small model on agent traces without writing training scripts?

Write a Contract Prompt that specifies your base model ID, trace dataset ID, parameter sweep ranges, hardware budget, Tracchio project, adapter push targets, selection rule, and eval benchmarks. Point a tool-use coding agent at a skills repo containing TRL, HF Jobs, and Tracchio workflows. The agent orients, writes a TRL script, runs the sweep on HF Jobs, and selects the best adapter by held-out loss.

How do I write a good Contract Prompt for the SFT agent?

State every constraint explicitly: base model HF ID, dataset HF ID, SFT algorithm, sweep ranges for learning rate, LoRA rank and sequence length, HF Jobs hardware tier and budget ceiling, one shared Tracchio project name, adapter push destination, the rule that the best run is selected by held-out eval loss, and eval benchmarks. Require the agent to orient itself before spending any budget.

How does agent-driven SFT compare to writing training scripts manually?

Manual scripting requires you to author TRL configs, masking logic, and job scheduling yourself, which is slow and error-prone. Agent-driven SFT moves you one abstraction level up: you define outcomes and constraints in a Contract Prompt and verify artifacts, while the agent handles implementation. Skills files encode tested workflows so the agent avoids common failure routes, making the pipeline repeatable and compounding over time.

When should I use SFT versus reinforcement learning for training an agent?

Use SFT when you want to imitate existing high-quality agent traces and focus a general model on a specific domain or harness format quickly. SFT teaches imitation — it cannot exceed the quality of its traces. Use RL and environment-based training when you need genuine agent capability improvement beyond the demonstrations. SFT is step one; RL builds on it.

What results can I expect from agent-driven SFT?

You get a focused, deployable model adapter that imitates your agentic traces, plus tracked metrics (train loss, eval loss, token accuracy, entropy) and benchmark eval scores on the model card. The model reproduces tool-call formatting and multi-turn behaviour from your traces. Expect closer imitation, not superhuman agent performance — held-out loss measures imitation quality, not task capability.

Why do I need to mask user-side tokens during SFT?

Masking user-side tokens with -100 labels is the defining act of SFT — it tells the loss function to ignore the user's instructions so only assistant completions contribute to weight updates. Without masking, the model learns to generate the user's turns instead of the assistant's responses, producing a broken fine-tune that predicts prompts rather than answers.

Why is sequence length important when training on agentic traces?

Agentic traces are long multi-turn sequences, so context window length and truncation strategy directly determine what the model can learn. If sequences are truncated too aggressively, the model never sees full tool-call chains or diffs. Always include sequence length in your parameter sweep alongside learning rate and LoRA rank — it's a first-class hyperparameter for agentic data.

How do I stop the agent from wasting money on failed training jobs?

Enforce the orientation phase before any paid HF Jobs launch: the agent must resolve the exact model ID, confirm the dataset exists and has the expected shape, verify push and job authorizations, and pass a cheap smoke-test run without OOM errors. If the agent skips orientation, intervene and add an explicit orientation rule to the Contract Prompt.

// GET THIS SKILL — FREE