How to Build a Legal AI Assistant That Cites Law
For Legal tech product builders · Based on Denisov Agent Specialization RAG vs Fine-Tuning Framework
// TL;DR
Legal assistant agents must do two hard things simultaneously: write in formal, jurisdiction-specific legal language with correct reasoning patterns, and cite current legislation that changes over time. This framework routes both. The Behaviour-Change Gate triggers YES on legal terminology and reasoning, so you fine-tune for style. The Data-Dynamism Gate triggers DYNAMIC because laws update, so you layer RAG over a live legal database with metadata filters for jurisdiction and document type. The result is a hybrid agent that never drifts into casual language — even in long conversations — while grounding every claim in current statutes with proper citations.
Why won't a general LLM work as a legal assistant?
Because a base model fails legal use cases in two distinct ways. First, it drifts into casual language and loses jurisdiction-specific reasoning patterns — a behaviour problem. Second, it cites outdated or hallucinated statutes because its training data is frozen and laws change — a data problem. Solving one doesn't solve the other, and confusing the two is the most expensive mistake in legal tech.
This framework separates them cleanly with two gates.
How do the two gates apply to a legal assistant?
Behaviour-Change Gate: Do you need to change tone, format, or reasoning patterns? Legal writing demands formal language, precise terminology, and jurisdiction-specific reasoning. That's a clear YES — fine-tune for style and legal terminology so the model never drifts into casual phrasing, even deep into a conversation where system-prompt instructions would otherwise be forgotten.
Data-Dynamism Gate: Is your knowledge dynamic? Legislation updates continuously, so the answer is DYNAMIC. Use RAG over a live legal document database. Critically, attach metadata filters for jurisdiction and document type at index time, so a query about California employment law never surfaces a Texas statute.
Both gates fire, so you build a hybrid architecture: fine-tuning locks in legal voice and reasoning; RAG ensures every citation refers to a current statute.
How do I make sure citations are accurate and current?
Citation accuracy is a RAG discipline, and it starts upstream — poor retrieval is almost never the retriever's fault.
1. Document processing: Parse and normalise statutes, case law, and regulations into a consistent format. Legal documents are dense; clean structure matters.
2. Chunking: Start with recursive character chunking. Only escalate to semantic or agentic chunking if you can demonstrate that clause boundaries are being split badly and hurting retrieval.
3. Metadata at index time: Attach jurisdiction, document type, effective date, and statute number. This lets you filter to current law in the correct jurisdiction rather than trusting semantic similarity to get it right.
Use Hybrid + Rerank retrieval: keyword search catches exact statute citations while semantic search captures conceptual matches. Retrieve ~100 candidates, rerank to the top 10. For multi-hop queries — a fact pattern requiring inference before searching — pre-process the query to extract explicit legal search intent first.
How do I fine-tune for legal language responsibly?
Garbage in, garbage out is amplified in law — training on sloppy or incorrect examples teaches the model bad reasoning permanently. Curate ruthlessly. Use an agentic data-preparation pipeline to convert vetted legal Q&A, memos, and briefs into clean training pairs, and generate paraphrased variants to improve generalisation. A thousand meticulously reviewed pairs beat a million scraped ones.
Choose LoRA as your default — it's cheap, fast, and keeps the base model frozen while adding a small adapter. That matters for legal tech where you may maintain separate adapters per practice area or jurisdiction.
After every retraining cycle, guard against catastrophic forgetting: validate against prior benchmarks so new training on, say, employment law doesn't degrade the model's contract-law reasoning.
What does the finished legal agent do well?
- Maintains formal legal language and jurisdiction-correct reasoning across long, complex conversations.
- Grounds every claim in current statutes, filtered by jurisdiction and document type.
- Provides citations that point to live, up-to-date law rather than frozen training data.
- Refuses to drift casual because the constraint lives in the weights, not just the prompt.
Next step: Map your agent's failure modes, run both gates (they nearly always both fire for legal use cases), and scope a hybrid build — starting with jurisdiction-tagged indexing and a LoRA fine-tune on rigorously curated legal writing.
// FREQUENTLY ASKED QUESTIONS
Should I fine-tune a legal AI on current legislation?
No — laws change, so legislation is dynamic data that belongs in RAG, not fine-tuning. Fine-tuning on statutes that update would leave the model citing outdated law. Instead, fine-tune only for legal language, terminology, and reasoning patterns, and layer RAG over a live legal database with jurisdiction and document-type metadata filters for current citations.
How do I stop a legal AI from writing in casual language?
Fine-tune it. Formal legal tone and jurisdiction-specific reasoning are behaviour constraints, and system-prompt instructions get forgotten on long conversations. Fine-tuning internalises the formal voice into the model's weights so it can't be context-flushed. RAG won't help here — it supplies facts, not style. Curate high-quality legal writing examples before training to avoid teaching bad habits.
How do I ensure legal citations reference the correct jurisdiction?
Attach jurisdiction and document-type metadata to every chunk at index time, then filter on it during retrieval. Without metadata filtering, the retriever resolves ambiguity semantically and may surface the wrong state's statute. Combine this with Hybrid + Rerank retrieval so exact statute-number matches from keyword search aren't lost to pure semantic similarity.