A sales demo of an AI tool for lawyers is easy to enjoy. It summarizes a long contract in seconds, drafts a clean letter and answers questions about a deposition. What the demo rarely shows is where your client's information goes once you click submit.
Before your firm signs anything, work through the five questions below. They apply to a research assistant, a drafting tool, a transcription service or an AI feature bundled into software you already use.
Question 1: Is Our Data Used to Train Models?
This is the question that matters most. Some consumer tools use inputs to improve their models by default. Business and enterprise tiers often exclude that, but terms differ.
- Get the answer in the contract or written terms, not just a sales conversation.
- Ask whether the setting can be changed by individual users or only by an administrator.
- Ask about both the vendor and any underlying model provider it relies on.
ABA Model Rule 1.6(c) calls for reasonable efforts to prevent unauthorized disclosure of client information, and Formal Opinion 512 discusses confidentiality in the context of generative AI. Informed consent from clients may be appropriate in some situations, and you should confirm what your state bar expects.
Question 2: Where Is Data Stored, and for How Long?
- Which country and which cloud provider hold our data?
- Are prompts and outputs retained, and for how long?
- Can we delete data on request, and is deletion confirmed?
- What happens to data if we cancel?
Some clients, particularly in regulated industries, restrict where their data may reside. Check outside counsel guidelines before you adopt a tool for their matters.
Question 3: Who Can See Our Information?
Ask about access by the vendor's employees, contractors and subprocessors.
- Is human review of prompts ever performed, such as for abuse monitoring?
- Which subprocessors are involved, and will we be notified if they change?
- Is data separated between customers?
- Are access logs available to us?
Question 4: What Security Controls and Assurances Exist?
You are not expected to audit a vendor's data center, but you can ask for evidence.
- Independent security reports, such as SOC 2 reports, and whether you may review them under confidentiality
- Encryption in transit and at rest
- Support for single sign-on and multi-factor authentication
- Role-based access, so not everyone sees everything
- A documented process for notifying customers of security incidents, with a stated time frame
If a vendor cannot or will not answer, treat that as an answer.
Question 5: What Do the Contract Terms Actually Say?
Marketing pages and contracts diverge. Read, or have counsel read:
- Ownership of inputs and outputs
- Indemnification and liability caps
- Breach notification obligations
- Termination rights and data return
- Whether the vendor may change terms unilaterally
What About Accuracy?
Security is only half the evaluation. Test the tool on real, non-confidential tasks and check the outputs. Generative AI can produce confident but wrong statements, including invented citations. Verification remains the lawyer's responsibility, and a vendor's claims about accuracy are not a substitute for your own checking.
Build a Simple Scorecard
Create a one-page form with these five questions and a place for answers, dates and who reviewed them. Keep it with your approved-tools list. When a client or insurer asks how you vet AI, you will have a dated record.
Do Not Forget the Tools Already Inside
Many firms focus on new AI products and miss features switched on inside existing software. Review settings in your email, document and meeting platforms, including AI note-takers that join calls automatically. Decide deliberately which are allowed.
Counsel Cyber helps law firms evaluate AI vendors and configure approved tools with sensible controls. If you have a product under consideration, we are glad to review it with you before you commit.