Jump to section
- 1. Where Does Client Data Go, and Who Can Access It?
- 2. Is My Data Used to Train the Model?
- 3. Where Are Servers Located, and What Law Governs the Data?
- 4. What Happens to My Data When I Cancel?
- 5. What Is the Accuracy Rate, and How Is It Measured?
- 6. What Is Covered by the Indemnity Clause?
- 7. How Does the Tool Handle Canadian Law Specifically?
- 8. What Is the Contract Term and How Do Prices Change?
- 9. Is There a Real Audit Trail?
- 10. Who Do I Call When Something Goes Wrong?
- How to Use This List
Legal AI vendors have multiplied quickly, and the sales pitches have gotten sophisticated. Every demo looks capable in a controlled environment. The question is whether the tool holds up in actual practice at a small Canadian firm, and whether buying it creates obligations or risks you did not plan for. These ten questions give you a structured way to evaluate any vendor before you commit to a trial, let alone a contract.
None of these questions are hostile. A vendor with a good product and honest terms will answer all of them clearly. The ones that hedge, deflect, or cannot answer are telling you something important.
1. Where Does Client Data Go, and Who Can Access It?
This is the foundational question. When you upload a document or type a query that includes client information, that data goes somewhere. Find out exactly where: which servers, which country, which cloud provider. Canadian law society guidance on cloud storage generally requires that you know where client data is processed and stored, and that you have assessed the risk.
Ask specifically whether support staff or engineers at the vendor can access your data, under what circumstances, and whether that access is logged. A good answer is specific. A bad answer is a redirect to a marketing page about "bank-level security."
2. Is My Data Used to Train the Model?
Many AI vendors use customer interactions to improve their models. That is a reasonable business practice for consumer tools. It is not an acceptable default for a law firm. Your client's documents and queries cannot be used as training data without informed consent, and obtaining that consent from clients for every interaction is not practical.
Look for a clear contractual commitment that your data is not used for model training, and check that this setting is actually active on the plan you are buying. Some vendors offer this commitment only on enterprise tiers. Know which tier you are on.
Watch out. "We may use anonymized data" is not the same as "we do not use your data." Anonymization is imperfect, and your law society's guidance does not carve out an exception for anonymized client information.
3. Where Are Servers Located, and What Law Governs the Data?
Data location matters for two reasons. First, some law societies have issued guidance recommending or requiring that client data be stored in Canada. Second, data stored in the United States may be accessible to US authorities under legislation like the CLOUD Act, regardless of contractual commitments. Ask for the physical server location and the governing law in the terms of service. If the answer is "AWS in us-east-1," that is not a Canadian server.
4. What Happens to My Data When I Cancel?
Vendor contracts often address data deletion on cancellation in the fine print. Some delete data within 30 days. Some retain it for 12 months or indefinitely unless you submit a specific deletion request. Know before you sign. Ask for the deletion timeline in writing, and ask whether you can export your data in a portable format before cancelling.
5. What Is the Accuracy Rate, and How Is It Measured?
Every legal AI vendor claims high accuracy. The useful follow-up is: accurate at what, measured how, and by whom? A vendor that cites an internal benchmark on a curated dataset is not telling you how the tool performs on real files from small Canadian practices. Ask whether they can share third-party evaluations, whether the accuracy figures cover your practice area and jurisdiction, and how the tool handles cases where it does not know the answer versus cases where it confidently gives a wrong one.
The post on reviewing AI output responsibly covers why confident wrong answers are the most dangerous kind. That context makes this question more specific: ask the vendor how the tool signals uncertainty, and ask to see examples.
6. What Is Covered by the Indemnity Clause?
Legal AI contracts almost universally disclaim liability for errors in the tool's output. That is expected. What matters is whether the vendor indemnifies you for anything: data breaches caused by their infrastructure, for example, or intellectual property claims arising from their training data. Read the indemnity and limitation of liability clauses before you sign. If you are not sure how to read them, that is a question for your firm's own legal review.
7. How Does the Tool Handle Canadian Law Specifically?
Most legal AI tools were trained primarily on US legal material. Canadian law is meaningfully different: bijural (common law and civil law), federal-provincial division of powers, its own statutes and case law. Ask directly whether the tool's training data includes substantial Canadian content, which jurisdictions it covers, and whether it flags the gap when a query falls outside its reliable range.
A tool that does not know what it does not know about Canadian law is more dangerous than one that simply says "I cannot answer that reliably." Ask for a demonstration with a question from your actual practice area and jurisdiction, and verify the answer yourself before you are impressed by how plausible it sounds.
8. What Is the Contract Term and How Do Prices Change?
Annual contracts with automatic renewal are common in SaaS. So are price increases at renewal. Ask about the minimum contract term, what notice is required to cancel, and whether there is a cap on annual price increases. Also ask about seat-based versus usage-based pricing and which model applies at the price point they are quoting. A practice that grows or shrinks by a few people should not face a billing cliff.
The general vendor evaluation framework in vetting a software vendor covers the contract and operational side of these questions in more depth.
9. Is There a Real Audit Trail?
Professional responsibility requires that you be able to account for work done on a matter. If an AI tool produces output that becomes part of a file, you need to be able to show what the tool produced, who reviewed it, and what changes were made. Ask whether the tool maintains an audit log of queries, outputs, and user actions. Ask whether that log is exportable. Ask how long it is retained.
This is not a hypothetical concern. A complaint, a malpractice claim, or a regulatory inquiry can arise years after a matter closes. Your records need to cover that window.
10. Who Do I Call When Something Goes Wrong?
Ask for the specific support path for billing errors, data concerns, and technical failures. Find out whether support is available during Canadian business hours, whether there is a dedicated account contact at your tier, and what the service level commitment looks like in the contract rather than in the sales brochure. A vendor that has not thought through their support structure for small firms is one that will frustrate you at the worst moment.
Tip. Ask the vendor for two or three Canadian small-firm references you can contact directly. A vendor confident in their product will provide them. One who cannot is showing you something.
How to Use This List
Send these questions to the vendor in writing before the demo, not after. Their willingness to answer in writing, before they have a signed contract, tells you a great deal about how they will behave after you are locked in. Answers that are clear and specific suggest a vendor that has thought through the professional context their tool operates in. Answers that are vague or redirect to marketing material suggest you should keep looking.
If you have already drafted an AI policy for your firm using the framework in writing an AI policy your small firm will actually use, these questions translate directly into the vendor-approval criteria that policy calls for. For a broader look at how AI tools fit into a well-run small practice, the legal tech and AI hub covers everything from triage automation to safe drafting automation. When you are ready to see how a purpose-built practice management system handles data security and audit trails, A1 CMS is worth a look.
