The AI Vendor Evaluation Scorecard

AI Vendor Evaluation Scorecard: Vendor Scoring Prompt

How to use: paste this entire prompt into an AI tool that can browse the web (ChatGPT, Claude, Gemini, or Copilot with web search turned on). Fill in the vendor details in the last section, then send it.

You are scoring a legal AI vendor against the AI Vendor Evaluation Scorecard. Your reader is a small law firm owner with no technical background. The results will be published under the name of a practicing attorney, next to the vendor's name, so accuracy matters more than speed. Every Pass must be backed by a source a reader could open and check for themselves, or by an answer the vendor gave the firm directly.

The one rule that governs everything

Score only what is publicly available before anyone signs a contract, plus anything the vendor has confirmed directly to the firm (see "Answers the vendor gave you directly" below). A Pass means the answer is written down somewhere a prospective customer can read without a sales call, an NDA, or a signature:

* the vendor's own website, product pages, pricing page, or help center
* the vendor's security page or Trust Center
* the vendor's published legal documents: Terms of Service, Platform Agreement, Master Subscription Agreement, Data Processing Addendum (DPA), Security Addendum, Business Associate Agreement (BAA), Privacy Policy, subprocessor list
* a current independent audit or certification the vendor publicly lists (SOC 2 Type II, ISO 27001, and similar)

These do NOT count toward a Pass:

* third-party reviews, blog posts, comparison sites, news articles, or other AI tools' summaries. You may use them to find leads, then confirm the lead on the vendor's own pages. If you cannot confirm it there, it does not count.
* anything a sales rep would say on a call, unless the firm owner provides it in the last section as an answer the vendor gave directly
* your own assumptions about what an enterprise vendor "probably" has

Answers the vendor gave you directly

If the firm owner pastes answers the vendor gave directly (in an interview, on a call, or in writing) into the last section, count them the same as a published source. Label each one in the report as "Confirmed by the vendor directly," with who said it and when, if given. Do not count anything the firm owner believes about the vendor but did not get from the vendor.

How to research

1. Find the vendor's official sources first. Search for the vendor's Trust Center, security page, legal or terms page, DPA, privacy policy, subprocessor list, pricing page, and help center. Fetch them; do not rely on search snippets.
2. Read every legal document the vendor publishes, not just one. Protections often sit in a different document than you expect. (Example: a no-training commitment missing from a Security Addendum may be in the Platform Agreement.) Before you mark anything "not found," confirm you checked every published legal document and list them in your report.
3. Score the specific AI product named in the request. If the vendor has several products with different terms, note which terms you applied.
4. Score what a small firm on the vendor's standard, self-serve, or entry plan would get. If a protection exists only on an enterprise tier or must be negotiated, mark it Fail and say "enterprise plans only" in the note.
5. Record the date you checked. Vendors change their terms.

Three kinds of Fail, and you must say which

* Not found: you checked every official source and the commitment is not there.
* Contradicted: the vendor's own sources say conflicting things (for example, a DPA says customer data is never combined, while investor or marketing materials say every customer's cases train the model). Quote both sides.
* Insufficient: the vendor says something on the topic, but it falls short of the Pass rule below.

A contradiction is more serious than silence. Always flag it.

The twelve questions

The first nine are Pass or Fail. Questions 3, 5, and 7 are hard stops. Questions 10, 11, and 12 are graded 1, 2, or 3.

BUILT: what's underneath?

Q1. Which large language model actually powers this (ChatGPT, Claude, Gemini), and what will they tell you before switching to a different one?

* Pass: the vendor itself names the underlying model or model provider on an official channel (for example OpenAI, Azure OpenAI, Anthropic, Google), AND some published document commits to notifying customers before material changes to the service. A general change-notice clause in the terms counts. No specific number of days is required.
* Fail: the vendor names nothing, or only says "proprietary AI" or "our own models" without saying what they are built on. A third party naming the model does not count.

Q2. Can you test the product without a long-term contract?

* The point of this question: can a firm owner actually get hands-on with the product, using it themselves, before they are locked in?
* Pass: an official source shows a prospective customer can get hands-on access to the real product without first signing an NDA or an annual or multi-year contract. A free trial, a sandbox, a pilot, a month-to-month plan, or a demo account the customer uses themselves all count. Having to go through a sales call or demo request first is fine and does not cause a Fail.
* Fail: hands-on access requires an NDA or an annual or multi-year contract first, or a pilot that already commits the firm to paid seats. Also Fail (not found) if nothing the vendor publishes says hands-on access is available at all, for example the site only says "Book a demo" and describes nothing further. In that case, the question to ask the vendor should be whether the firm can test the product hands-on before signing an NDA or an annual contract.
* A demo where the vendor only shows the product on screen, and the customer never uses it themselves, is not hands-on access.
* Always say whether a free trial specifically exists, and its length if stated.

RISK: what happens to data?

Q3. HARD STOP. Are your inputs and outputs used to train either the platform itself or the underlying model it runs on?

* Pass: an official, published source states that customer inputs and outputs are not used to train the vendor's own models AND are not used to train the underlying provider's models (for example, a no-training clause that also binds subprocessors, or a stated zero-data-retention or no-training agreement with the model provider). Both halves are required.
* Use of de-identified, anonymized, or aggregated customer data (for example, to improve the service or check quality) is acceptable and does not cause a Fail. If marketing says "we never use your data" while the contract allows only de-identified or aggregated use, that is not a contradiction for scoring; mention it neutrally as something to be aware of.
* Fail: silence on either half; training on your actual, identifiable client content, including a license to use the content itself to train or improve AI; training that is on by default with an opt-out; or a genuine contradiction between the vendor's own sources about identifiable data.

Q4. How long is your data kept, how is it deleted, and how do you verify it's gone?

* Pass: an official source names at least one specific mechanism, such as deletion within a stated number of days of your request, deletion of backups on a fixed schedule, cryptographic erasure (destroying the encryption key), or a certificate of destruction you can request.
* Fail: "per our data retention policy," "industry best practices," or a deletion promise with an undefined exception (for example, "except as retained under our policies" when those policies are not published).

Q5. HARD STOP. Do they have a SOC 2 Type II audit (and a BAA if HIPAA applies)?

* Pass: the vendor's own official sources say it has a SOC 2 Type II audit, or SOC 2 with the type not stated. Assume the audit is current and covers the product. Do not fail a vendor because the audit date shown is old, no date or auditor is named, or the report must be requested through a Trust Center. Websites often lag behind the latest report.
* A DPA is not required to pass. If the vendor has one, the firm should ask for it before signing; note whether one is published or available on request.
* A BAA is required only if the firm will give the vendor health information covered by HIPAA. Most law firms, including personal injury firms, are not HIPAA covered entities, so a missing BAA does not cause a Fail on its own. Note whether a BAA is available, and remind a firm that acts as a business associate of a healthcare provider to ask for one.
* Fail: only a SOC 2 Type I, or a Type II that is still "in progress"; wording like "SOC 2 ready" or "aligned with SOC 2," which describes readiness, not a completed audit; no SOC 2 claim at all (ISO 27001 alone does not substitute; note it if present); or HIPAA-covered health information with no BAA available.

Q6. Who are the sub-processors, will they notify you before adding one, and where does the data live?

* Pass: all three are published: a subprocessor list, a commitment to give notice before adding a new subprocessor, and where customer data is stored.
* Fail: any of the three is missing.

Q7. HARD STOP. Can one client's data ever surface in another client's session?

* Pass: an official source states customer data is isolated or logically separated between customers, AND an independent audit or test supports it. A SOC 2 Type II (or SOC 2 with the type not stated) that the vendor says it has counts as current support, as in Q5. A published penetration test or a red-team exercise the vendor describes also counts.
* Fail: no isolation statement at all; or an isolation statement with no SOC 2 claim (or only a Type I) and no penetration test or red-team exercise behind it.

Q8. When there is a data breach, how soon will you be notified, and who is the contact?

* Pass: a published document commits to notifying the customer within 72 hours or fewer. A named individual contact is NOT required; record any contact or channel given.
* Fail: no specific time; "without undue delay" or "promptly" with no number; a window longer than 72 hours (treat "3 business days" or longer as over 72 hours).

WORTH: the price, and the risk.

Q9. If you leave, do you get your data back in a usable format, with written proof of destruction?

* Pass: an official source describes how a customer exports their data in a usable format (CSV, PDF, native files, or similar) AND states the vendor deletes or destroys it afterward (a destruction certificate, or a stated deletion commitment after termination).
* Fail: either half is missing.

Q10. Graded 1 to 3. Does a person at the vendor's company need your permission before they can look at your data, and is that access logged and revocable?

* 3: an official source states vendor staff need the customer's explicit permission before accessing customer data, and that access is logged and can be revoked.
* 2: restricted or logged staff access is described, but permission and revocation are not both stated.
* 1: nothing public on vendor staff access.

Q11. Graded 1 to 3. Is there a full, exportable record of what the AI itself was asked and what it answered?

* 3: an official source states there is a log of every AI prompt and answer, tied to the user, that the customer can export.
* 2: a general AI activity log or oversight feature is described, but it is not confirmed to cover every prompt and answer, or not confirmed exportable.
* 1: nothing public on AI activity logging.

Q12. Graded 1 to 3. Can the AI take actions on its own, and is there a kill switch you control?

* 3: an official source states that a person reviews and approves AI output before it reaches a client, a court, or the record; or, for tools built to act on their own, publishes what the AI can do without approval and a way for the customer to stop it.
* 2: approval gates or a stop mechanism are described for some features but not all, or only in general terms.
* 1: the AI acts on its own (sends messages, contacts clients, files, or runs scheduled tasks) and nothing public describes approval or a way to stop it.

Verdict

* WALK: any hard stop (Q3, Q5, Q7) is a Fail.
* HOLD: all hard stops pass, but any of Q1, Q2, Q4, Q6, Q8, Q9 is a Fail.
* PROCEED: all nine Pass. List any of Q10 to Q12 scored below 3 as the needs-work list.

Writing rules

* Plain English for a law firm owner. No jargon without a short explanation.
* Factual and neutral. Describe what the documents say and do not say. Do not speculate about motive.
* Quotes from vendor documents: 25 words or fewer each.
* Do not use em dashes anywhere in your output.
* If you are unsure, mark it Fail (not found) and say what to ask the vendor. Do not guess a Pass.

BEFORE YOU START

If you cannot open and read web pages, stop and say so plainly. Do not score from memory. Tell the reader: "I can't browse the web, so I can't check this vendor's documents. Please use an AI tool with web browsing turned on."

OUTPUT

Write a report a busy law firm owner can read in five minutes. Use plain English, short paragraphs, and no code or technical formatting. Follow this structure:

1. Headline: the vendor name, the product, and the verdict (WALK, HOLD, or PROCEED) in one line.

2. What this product does: two or three sentences in plain English.

3. What I checked: the date you checked, and a list of every vendor document you reviewed, each with its link and its last-updated date if shown. If the firm owner provided answers the vendor gave directly, list those too.

4. Verdict in one paragraph: why this vendor got this verdict.

5. Scorecard table with three columns: Question, Result, One-line reason. Include all twelve questions in order. Mark Q3, Q5, and Q7 as hard stops. Show Pass or Fail for Q1 to Q9 and 1, 2, or 3 for Q10 to Q12.

6. The details: for each of the twelve questions, write a short paragraph explaining what the vendor's documents say or don't say. Include a short quote (25 words or fewer) and the link to the page it came from for every Pass, and wherever a quote makes a Fail clearer. If a Pass rests on an answer the vendor gave directly, say "Confirmed by the vendor directly" instead of a link. For each Fail, say whether the answer was not found, contradicted, or insufficient, in plain words. Say how confident you are (high, medium, or low).

7. Contradictions: list any places where the vendor's own documents disagree, quoting both sides with links. If there are none, say so.

8. What to ask the vendor: a numbered list of plain-English questions the firm owner can copy into an email, one for each Fail and each graded question below 3.

9. A closing note: "This evaluation is based on the vendor's public documents as of [date], plus any answers the vendor gave directly, which are labeled. Vendors change their terms, so confirm anything important in writing before you sign."

THE VENDOR TO SCORE

Vendor: [vendor name]
Product to score: [product name]
Vendor website: [website address]
Today's date: [today's date]
Answers the vendor gave you directly (optional): [paste them here, with who said it and when, or leave blank]