Legal technology vendors now add artificial intelligence to nearly everything: research platforms, document review tools, intake chatbots, transcription services and practice management suites. Some of these tools can save real time. Every one of them also receives your data, and the terms that govern that data differ widely between products and between plans of the same product.
ABA Formal Opinion 512, issued in July 2024, discusses a lawyer's duties of competence, confidentiality and supervision when using generative AI, and it recommends that lawyers understand how a tool handles information before relying on it. The questions below translate that into a practical vendor review for administrators. Your state bar may have additional guidance.
Start with the use case
Decide what task the tool will perform and what data it will see. Summarizing public statutes is a different risk than analyzing a client's medical records. Defining the use case first lets you scale your scrutiny and may reveal that a lower-risk approach exists.
Questions about data
- Where does our data go? Is it processed in the vendor's environment, or passed to a third-party model provider? Where are the servers located?
- Is our data used to train or improve models? Ask for the answer in writing for your specific plan, since consumer and enterprise tiers often differ.
- How long is data retained, including prompts, uploaded files and outputs, and can we delete it on request?
- Who at the vendor can access our content, and under what circumstances, such as support or abuse monitoring?
- Is our data segregated from other customers, and how is it encrypted in transit and at rest?
- What happens at termination? Will data be returned and deleted, and will deletion extend to backups and to any subprocessors?
Questions about security
- Does the vendor hold an independent security attestation, such as a SOC 2 report, and will it share the report under a confidentiality agreement?
- Does it support MFA and single sign-on, and role-based access?
- How does it handle security incidents, and what is its notification commitment?
- Who are its subprocessors, and how are they vetted?
- Does it provide audit logs showing who used the tool and what they submitted?
Questions about accuracy and limits
Generative tools can state falsehoods confidently. Ask the vendor:
- How does the tool cite its sources, and can users verify them easily?
- What known limitations or failure modes does the vendor disclose?
- Is there a way to limit outputs to a trusted set of documents?
- What testing or benchmarks can it share, and how were they conducted? Treat vendor claims as marketing until you test the tool on your own work.
Questions about contracts
Have counsel review the agreement. Look for:
- Confidentiality terms that cover your inputs and outputs.
- A commitment not to use your data to train general models, if that matters to you.
- Breach notification timelines and cooperation obligations.
- Indemnification and liability caps that reflect the risk.
- Rights to change terms. A vendor that can alter data practices on short notice should make you cautious.
- Clear ownership of outputs.
Run a pilot before you commit
Select a small group and a defined set of non-sensitive or consented tasks. Compare the results with human work. Record errors found and time actually saved. Have a supervising attorney review outputs, consistent with Rules 5.1 and 5.3. Decide in advance what result would cause you to stop.
Watch for AI features that arrive by update
Existing software may add AI capabilities that are on by default. Ask vendors for notice of such changes, and review admin settings after major updates. Assign someone to check release notes quarterly.
Document your decision
Keep the questionnaire, the vendor's answers, your pilot results and the approval, with dates. That record supports your competence and supervision efforts and answers client and carrier questions about your AI practices.
Red flags
- Reluctance to put data handling in writing.
- No clear answer on model training.
- No security attestation or willingness to discuss one.
- Pressure to sign quickly or to skip a pilot.
- Claims of perfect accuracy.
Next steps
Counsel Cyber helps law firms evaluate AI tools and configure them securely, from reading data terms to setting up access controls. If you have a vendor under consideration, we can review it with you.