New AI products arrive every week, and each promises faster research, better drafting or automatic summaries of discovery. Some will be genuinely useful to a law firm. But a tool that sounds impressive in a demo can still create a confidentiality problem if you do not know what happens to the data you give it.
ABA Formal Opinion 512 on generative AI emphasizes that lawyers must understand the tools they use and protect client information. This post gives you ten questions to ask any vendor before approval. Keep the answers on file, in writing, so you can show your diligence later.
The ten questions
1. Will our data be used to train your models?
Many vendors offer different terms for consumer and business versions. Get the answer in the contract, not only in marketing copy. Look for a clear statement that your inputs and outputs are not used to train models for other customers.
2. How long do you retain our prompts, documents and outputs?
Ask about the default retention period, whether you can configure it, and whether you can request deletion. Also ask whether data is retained for abuse monitoring and who can see it.
3. Who at your company can access our data?
Understand whether human reviewers can read content, under what conditions, and how that access is logged and controlled.
4. Where is the data processed and stored?
Ask about geographic location and about subprocessors, meaning other companies that handle your data on the vendor's behalf, such as cloud hosting or model providers. Request a list and notice of changes.
5. How is the data protected?
Look for encryption in transit and at rest, strong access controls, MFA, and logging. Ask whether the vendor has an independent security attestation, such as a SOC 2 report, and request a copy under nondisclosure.
6. How do you handle a security incident?
Ask how quickly the vendor will notify you of an incident affecting your data and what support they provide. You may need that timeline to meet your own obligations.
7. How does the tool connect to our other systems?
Integrations with email, document management or practice management may grant broad access. Ask what permissions are required, whether they can be limited to specific folders or matters, and how to revoke access.
8. What controls do administrators have?
Look for single sign-on, role-based permissions, the ability to disable features, audit logs of user activity, and the ability to enforce firm-wide settings.
9. What are the contract terms?
Review confidentiality language, data ownership, indemnification, liability limits, and what happens to your data when the contract ends. Your own state bar's guidance and your clients' outside counsel guidelines may impose additional requirements.
10. How accurate and explainable is the output?
No vendor can promise error-free results. Ask how the product reduces fabrication, whether it cites sources you can verify, and what testing the vendor has done on legal tasks. Plan to verify everything yourself.
How to run the evaluation
- Designate a small group, such as a partner, the administrator and IT, to review requests.
- Use a standard questionnaire containing the ten questions above.
- Test the tool with non-confidential sample data first.
- Decide on the approved use cases and write them down.
- Add the tool to the approved list in your AI policy and train users.
- Review annually and when the vendor's terms change.
Red flags
- Reluctance to put data-use promises in writing
- No clear answer on retention or deletion
- Consumer-grade accounts for business use
- No administrative controls or logging
- Vague descriptions of security, such as "bank-level security" without specifics
Do not forget the tools already inside your software
AI features are being added to products you already use, including email, document tools and practice-management systems. They may be enabled by default. Ask your vendors what has changed and review the settings.
How Counsel Cyber can help
We assist firms with AI vendor reviews, secure configuration and policy development, so partners can say yes to useful tools with confidence. Contact us to set up a review process.