A new AI product arrives in your inbox every week, each promising faster research, drafting and summarization. Some are well built. Others are thin wrappers around a model with unclear data handling. Before any client information touches one of them, a firm should decide whether the vendor deserves that trust.
ABA Formal Opinion 512, issued in July 2024, discusses a lawyer's duties of competence, confidentiality, communication and supervision when using generative AI. It also notes that lawyers should understand how a tool uses the information entered. This checklist turns that principle into questions you can ask. It is general information, not legal advice.
Start with the use case
Define what you want the tool to do and what data it will see. A tool used to polish a public marketing blurb needs far less scrutiny than one asked to summarize deposition transcripts. Decide in advance the most sensitive data you would allow, because that sets the bar for diligence.
Data handling questions
- Training. Does the vendor use your inputs or outputs to train its models or those of third parties? Can you opt out in writing, and is that the default?
- Retention. How long are prompts, files and outputs stored? Can you delete them, and does deletion also reach backups?
- Location. Where is data processed and stored?
- Human review. Do vendor employees ever read your content, and under what conditions?
- Subprocessors. Which other companies, including model providers, touch the data?
Get answers in the contract or a written statement, not just on a marketing page, since web pages change.
Security questions
- Does the tool support single sign-on and multi-factor authentication?
- Can you assign roles and limit who may use it?
- Is data encrypted in transit and at rest?
- Are there audit logs of who did what?
- Has the vendor undergone an independent security assessment, and can you review the summary or report? Ask about the scope and the date, since a report covering only part of the service is less useful.
- How does the vendor handle vulnerabilities and incident notification, and how fast will it tell you?
Contract questions
- Does the agreement include confidentiality obligations that fit your duties to clients?
- Who owns inputs and outputs?
- What happens to your data if you cancel or the vendor is acquired?
- Is there indemnification, and what limits apply?
- Are there terms allowing the vendor to change data practices unilaterally?
Have counsel review the agreement. Click-through terms for free or low-cost plans are the most likely to be unsuitable.
Accuracy and reliability
Test the tool on non-sensitive material that mimics your work. Ask:
- How often does it produce wrong or invented content?
- Does it cite sources you can verify?
- Can it be steered with firm-specific templates?
- Does it behave consistently across runs?
Remember that verification by a lawyer remains necessary no matter how good the demonstration looked.
Embedded AI in existing tools
Vendors are adding AI features to products you already use. Treat each new feature as a mini-vendor review. Check settings, since features may arrive enabled by default, and confirm whether they change how your data is processed.
Document and decide
Record the review: who did it, what was asked, the answers, risks accepted and the approval decision. Add approved tools to a firm list and note conditions, such as "no client-identifying data" or "attorney review required." Revisit the review annually and when the vendor changes its terms.
Red flags
- Vague answers about training on your data
- No business or enterprise plan
- Refusal to provide security documentation
- Pressure to decide quickly
- No way to delete data
- Claims of accuracy with no explanation
Roll out in stages
Start with a small pilot group, set rules for the data allowed, collect feedback and expand slowly. Pair the rollout with training on what is permitted and why.
How we can help
Counsel Cyber helps law firms run vendor reviews for AI tools, configure access controls and write clear guidance for staff. If you have a tool under consideration, we are glad to walk through these questions with you before client data is involved.