New legal AI products appear constantly: research assistants, drafting tools, contract review platforms, intake bots and summarization services. Vendor pitches often highlight time savings and sophisticated models. Fewer pitches dwell on what happens to the confidential information a lawyer feeds into the tool.
A structured due diligence process helps you compare products fairly and document your decision. ABA Formal Opinion 512, issued in July 2024, discusses lawyers' responsibilities when using generative AI, including competence, confidentiality, supervision and fees, and encourages lawyers to understand the tools they use. This list translates that spirit into questions you can put to a vendor. Confirm state-specific guidance with your state bar.
Start with the use case
Before looking at products, define the job. Is it legal research, first drafts, contract review, document summarization or client intake? Different uses carry different risks. A tool that touches privileged client documents warrants stricter review than one that drafts marketing copy.
Confidentiality and data use
These questions matter most.
- Is our data used to train or improve the vendor's models, or the models of any third party? Can we contractually prohibit it?
- Do the vendor's underlying model providers receive our data, and under what terms?
- How long are inputs, outputs and logs retained? Can we delete them on request, and how is deletion verified?
- Who at the vendor can access our content, and under what circumstances?
- Is our data logically separated from other customers?
- Where is data processed and stored?
Get answers in writing, and compare them to the contract. Marketing claims are not commitments.
Security
- Does the product support single sign-on, MFA and role-based access?
- Is data encrypted in transit and at rest? Who holds the keys?
- What independent security audits or attestations does the vendor have? Ask for current reports and read the scope.
- Is there an audit log we can review?
- How does the vendor handle vulnerability management and penetration testing?
- What is the incident response and customer notification process, and how fast will we be told about a breach?
- How does the vendor vet its own subcontractors?
Accuracy and reliability
- How does the vendor test and measure accuracy for legal tasks, and can it share results and limits?
- Does the tool cite sources, and can users click through to verify them?
- How does it handle jurisdictional differences and recent changes in law?
- What happens when the tool is uncertain? Does it say so or guess?
- Is there a way to give feedback and see how issues are fixed?
Remember that no tool removes the lawyer's duty to verify. Courts have sanctioned lawyers for filing fabricated citations, so build a verification step into any workflow.
Fit with ethics and firm policy
- Does the vendor understand lawyers' confidentiality and privilege concerns? Ask how it supports ethical walls and matter-level separation.
- Can the tool be configured to meet client restrictions on AI use?
- Does it let administrators control which features are enabled?
- How will outputs be reviewed and supervised under our policy?
Contract terms
Have counsel review the agreement, and look for:
- Confidentiality commitments and data ownership terms.
- Restrictions on the vendor's use of your content.
- Breach notification obligations and timelines.
- Indemnification and limitation of liability, and whether they are realistic given the risk.
- Subcontractor and subprocessor disclosures, with notice of changes.
- Term, renewal and termination, and what happens to your data on exit.
- Rights to change the product or terms, and how you are notified.
Pricing and total cost
Look beyond the license: implementation, training, integrations, usage-based charges and the internal time required to review outputs. Consider how fee arrangements will reflect the tool. Opinion 512 touches on billing, indicating that lawyers should bill for time actually spent.
Business viability
Newer vendors may be acquired, change direction or shut down. Ask about funding, customer base and what happens to your data if the company is sold or closes.
Pilot before committing
Run a limited trial with non-sensitive or synthetic material, with a defined group and success measures. Compare outputs against lawyer-produced work. Record errors and the effort needed to correct them. Involve IT early to test integration and access controls.
Document the decision
Keep a one-page record per vendor: the use case, the answers received, the reviewers, the risks accepted and the approval. This record supports your AI policy, client questionnaires and your own future reviews.
Re-review periodically
Revisit approved tools at least annually and whenever the vendor changes its terms or architecture. AI features also arrive inside existing products, so check for new ones after updates.
Where Counsel Cyber fits
We help firms run vendor due diligence for AI and other cloud tools, including security review and configuration. If you are comparing products, we can help you build and score a question list.