Frequently Asked Questions About Hugging Face Agent-Driven SFT Training Framework
22 answers covering everything from basics to advanced usage.
// Basics
What does SFT actually teach a model to do?
SFT teaches a model to imitate the assistant completions in a trace dataset. It's a continuation of pre-training where the model predicts assistant responses from prompt-completion pairs, with user-side tokens masked. The training objective is to produce the best imitator of the traces — not the best agent. It cannot produce outputs better than the demonstrations it learns from.
What is the orientation phase and why is it mandatory?
The orientation phase is a pre-spend verification stage where the agent confirms the correct model ID resolves, the dataset exists with the expected shape, push and job authorizations work, and a cheap smoke-test job succeeds without OOM. It's mandatory because launching paid HF Jobs without it wastes budget on runs that will fail for avoidable reasons.
What is a smoke test in this workflow?
A smoke test is a minimal, cheap HF Job run executed during orientation to confirm infrastructure, data formatting, and authorizations work before the full parameter sweep launches. It catches OOM errors, malformed chat templates, and permission failures early, preventing expensive multi-job sweeps from failing partway through.
What is Tracchio and why must all runs share one project?
Tracchio (track.io) is a lightweight, free experiment tracking tool that logs live metrics from each HF Jobs training run and can be self-hosted on Hugging Face Spaces. All runs in a sweep must share one project so their loss, token accuracy, and entropy curves can be compared side by side, enabling proper selection of the best adapter.
// How To
How do I collect and prepare agentic traces for training?
Gather multi-turn conversations from your own agent sessions (any harness, any model) or source a public dataset of real sessions. Ensure each trace has user instructions, assistant tool calls, tool results, and continuations. Scrub secrets, API tokens, and credentials with a tool like Presidio HF before pushing the cleaned dataset to HF Hub.
How do I set up the skills repo for the agent?
Create or clone a working directory with skill files covering the SFT workflow (stages, data format, masking), TRL usage (SFTConfig and SFTTrainer patterns), HF Jobs (scheduling, smoke-testing, retrieving results), and Tracchio observability (consistent logging). Install skills via the TRL CLI or HF CLI where possible, and add an entry for any recurring agent mistake you want to prevent.
How do I validate the training script the agent writes?
Read the agent-generated TRL script and confirm it applies the chat template via apply_chat_template, masks user-side tokens with -100 labels, logs to the correct Tracchio project, and pushes each adapter to the specified HF repo. You don't need to write it, but you must be able to read and validate these four things before jobs run.
How do I run evals on the final fine-tuned model?
Trigger eval jobs via HF Jobs using Inspect AI or an equivalent framework. Run at least one domain-relevant benchmark or held-out task set and one general capability benchmark like HumanEval or MBPP. The agent records scores and adds them to the model card README in a summary table with job IDs, Tracchio links, adapter repo links, and per-benchmark scores.
// Troubleshooting
My training loss barely moves — what should I do?
Check Tracchio to see whether loss was still decreasing at the end of the run before concluding SFT doesn't work. If loss plateaued too early, consider running more steps, raising the learning rate, or increasing sequence length. Too few steps or too narrow a learning rate sweep is a common reason people wrongly conclude SFT failed.
My train and eval loss are diverging — what does that mean?
Divergence between train and eval loss means the model is overfitting — it's memorizing the training traces rather than generalizing. Reduce the number of steps or increase data diversity. Healthy training shows eval loss tracking train loss closely; when eval loss rises while train loss falls, stop training earlier or add more varied traces.
The agent launched jobs before verifying anything — how do I fix this?
Intervene immediately and add an explicit orientation rule to the Contract Prompt requiring the agent to resolve the model ID, confirm dataset shape, verify authorizations, and pass a smoke test before any paid job. Record this as a skill entry so future runs enforce orientation automatically. Skipping orientation is the most expensive avoidable failure in this workflow.
My fine-tuned model got worse at general coding — why?
You likely evaluated only on a domain benchmark and missed capability regression. Fine-tuning heavily on narrow traces can erode general abilities. Always run a general capability benchmark like HumanEval or MBPP alongside your domain benchmark so regressions are detected. If general scores dropped, reduce steps, add diverse data, or lower the LoRA rank.
The agent silently dropped a failed run — is that a problem?
Yes. Every run in the sweep should appear in the same Tracchio project, and failed runs should be diagnosed and retried, not dropped. Silently dropped runs can skew your best-adapter selection and hide systematic errors like OOM or template mismatches. Verify the agent retries failures and add a skill entry requiring failed runs to be surfaced.
// Comparisons
How does SFT compare to full RL training for agents?
SFT is imitation — it makes the model reproduce existing traces and cannot exceed their quality. RL and environment-based training let the model improve beyond demonstrations by optimizing against a reward or environment. SFT is faster, cheaper, and step one of the pipeline; RL is the more capable but expensive next stage. Use SFT to focus a model, RL to make it genuinely better.
How does agent-driven SFT differ from a generic fine-tuning tutorial?
Generic tutorials have you hand-write TRL configs, masking logic, and job scheduling for a single run. Agent-driven SFT abstracts this into a Contract Prompt where a coding agent handles implementation across a full parameter sweep, tracks everything in Tracchio, selects by held-out loss, and updates skills files to compound knowledge. You operate as a verifier, not a script author, making the process repeatable.
Is held-out loss a good measure of agent quality?
No — held-out loss measures imitation quality, not agent capability. A low loss means the adapter closely reproduces the trace completions, not that it's a capable autonomous agent. Don't over-interpret low loss as proof of a strong agent. Use held-out loss to select the best imitator, then rely on task-based evals and RL for genuine capability assessment and improvement.
How does LoRA compare to full fine-tuning in this framework?
LoRA (PEFT) attaches small trainable adapter matrices to a frozen base model, making runs cheaper and faster while keeping only adapters until the best is selected. Full fine-tuning updates all weights and is more expensive. This framework sweeps LoRA rank as a hyperparameter, pushes adapters per run, and only pushes full weights for the winning run selected by held-out loss.
// Advanced
How do I turn the agent's own orchestration trace into a training asset?
After each run, save the full agent trace as an HF bucket or dataset. Review it to find routes that wasted compute or caused errors, then add entries to the relevant skill files to prevent repetition. Over time this compounds — the skills repo becomes living documentation of your best-known SFT workflow, and the agent's traces can even feed future fine-tuning.
How should I handle harness-specific chat template formatting?
Specify the target harness's chat template format as an explicit constraint in the SFT workflow skill. The agent's script applies it via apply_chat_template and masks all user-turn tokens. Sweep sequence length aggressively because harness traces are long, and use a harness-specific eval — can the model complete tasks in that harness? — as the primary benchmark. Watch entropy decrease in Tracchio as evidence of convergence.
What metrics indicate healthy SFT training in Tracchio?
Train loss decreasing confirms the model is learning completions; eval loss tracking train loss confirms no significant overfitting; token accuracy increasing and entropy decreasing show convergence; and the learning rate schedule should be high early and tapering. If any of these misbehave — flat loss, diverging eval loss, rising entropy — investigate before drawing conclusions about the run.
How do I scrub secrets from traces before publishing?
Run a scrubbing tool like Presidio HF over your trace dataset to detect and redact API tokens, credentials, and other sensitive values before pushing to any public HF Hub repo. Real agent sessions frequently capture secrets in tool calls or outputs, so treat scrubbing as a required step — never push raw traces to a public dataset.
Can I use any coding agent, or only a specific one?
You can use any tool-use capable coding agent — Codex, Claude, or similar — as long as it can read your skills repo and execute HF CLI commands. The skills files and Contract Prompt do the heavy lifting by encoding tested TRL, HF Jobs, and Tracchio workflows, so the framework is agent-agnostic. Point the agent at the skills repo and send the Contract Prompt.