Generative AI tools can draft, summarize and brainstorm at impressive speed. They can also invent. A hallucination, in this context, is output that sounds authoritative but is partly or wholly wrong, including cases that do not exist, quotations that were never written and statutes that say something different from what the tool claims.
Courts have addressed filings containing fabricated citations produced by AI tools, and some have imposed sanctions or adopted rules requiring disclosure or certification. The lesson for firms is not to avoid the tools entirely but to build verification into the workflow, so that checking is routine rather than dependent on one careful person.
What the Ethics Guidance Says in General
ABA Formal Opinion 512, issued in July 2024, discusses generative AI in light of the Model Rules, including competence, confidentiality, communication, candor toward the tribunal, supervision and fees. In substance, it indicates that lawyers should understand the capabilities and limitations of the tools they use, and that responsibility for the work product remains with the lawyer. Individual states and courts may add their own requirements, so check yours.
Why Hallucinations Happen
These systems generate text by predicting likely words, not by looking up verified facts. Unless a tool is specifically connected to a trusted legal database, it has no built-in way to confirm that a citation is real. Even tools that cite sources can misstate what the sources say. The fluency of the writing makes errors easy to miss.
A Verification Workflow
Step 1: Label AI-assisted work
Require anyone using AI to draft or research to mark the document internally, so reviewers know extra checking is needed. This is not about blame. It is about directing attention.
Step 2: Verify every authority independently
For each case, statute, regulation or secondary source mentioned:
- Locate it in a trusted database such as your legal research service.
- Confirm that it exists, with the correct name, court, date and citation.
- Read the relevant portion and confirm it supports the stated proposition.
- Check that it is still good law.
- Verify direct quotations word for word.
Step 3: Check facts against the record
Confirm that factual statements match source documents, deposition transcripts, contracts or exhibits. AI summaries sometimes blend or reverse details.
Step 4: Second-person review
For anything going to a court or opposing party, have someone other than the author review the verification. Supervising attorneys under Rules 5.1 and 5.3 bear responsibility for reasonable measures to ensure compliance, so a documented second review is a practical safeguard.
Step 5: Document the check
Keep a brief record: who verified what, when and with which sources. It need not be elaborate. A checklist stored in the matter file is enough.
Choose the Right Tool for the Task
- Brainstorming and outlining: lower risk, but still verify before relying.
- Summarizing documents you supply: moderate risk, so spot-check against the source.
- Legal research and citations: highest risk with general-purpose tools. Prefer legal-specific products that link to real sources, and still verify.
Court Rules and Disclosure
Some courts and judges have standing orders addressing AI use in filings. Before filing, check the local rules and the assigned judge's procedures. Build this into your filing checklist.
Training Points for Staff
- AI output is a draft, not an authority.
- Never paste a citation into a brief because a tool provided it.
- Ask the tool for its sources, then verify them yourself, and be aware it may invent those too.
- Report errors you find, so the firm can learn which uses are risky.
Red Flags in Output
- Citations that are suspiciously perfect or fit the argument too neatly.
- Case names you cannot find in any database.
- Quotes lacking pinpoint citations.
- Confident statements about recent developments the tool cannot know.
Fees and Client Communication
Consider how AI-assisted work is billed and whether to inform clients, consistent with the guidance in Opinion 512 and your engagement terms. Verification time is real work and should be accounted for honestly.
How Counsel Cyber Helps
Counsel Cyber helps firms put governance around AI use, including approved-tool lists, policies and training that make verification a habit. If you would like a one-page verification checklist adapted to your practice areas, we can help you draft it.