Why does a vendor demo tell you little about an AI tool?
A demo runs on data the vendor chose. Your records are messier. NIST's Generative AI Profile (NIST AI 600-1, July 2024) notes that "measurement gaps can arise from mismatches between laboratory and real-world settings." It also advises organizations to avoid extrapolating performance from "narrow, nonsystematic, and anecdotal assessments." A demo is one of those.
Business tasks remain hard for these systems. Salesforce AI Research built a test of nine language-model agents on CRM work from synthetic company data and released it as a preprint in May 2025. The best agents finished roughly 58% of single-turn tasks, and success fell to roughly 35% once a task stretched over several turns. The agents also showed "near-zero inherent confidentiality awareness." This is vendor research on synthetic data, using models from early 2025, and later models may do better. Only your own data will give you your own number.
What questions should you ask when evaluating AI tools for a revenue team?
Some of these answers come from the vendor and some from your own team. The answers from your own team tell you more.
-
What will it read, and where does our data go?
A tool that reads your CRM and your inbox holds your customer list and your pricing. Ask which systems it connects to and which other companies handle the data for the vendor. The ICO, the UK's data protection regulator, says that under UK GDPR a processor must not engage a sub-processor without the controller's "prior specific or general written authorisation." Listen for a written list of every system and sub-processor, and a plain answer on whether your data trains any model.
-
How does it do on our own data, against answers we already know?
Questions about last year's won and lost work already have settled answers, so put them to the tool and compare. NIST's profile recommends evaluating generative AI "in real-world scenarios" to reveal issues "that might not surface in controlled and optimized testing environments." Listen for a vendor who welcomes the test. Look hardest at the confident wrong answers, because a seller will repeat them to a customer.
-
Who will use it in a normal week, and what will they stop doing?
A tool earns its cost in an ordinary week, long after launch. In Pew Research Center's October 2024 survey of 5,273 U.S. workers, 55% said they rarely or never used AI chatbots at work, and about one in ten used them every day or a few times a week, though use may have grown since. Listen for named people, and for the report or habit the tool replaces in their week. If nothing stops, the tool adds work.
-
How will we know when its answers get worse?
Answers can drift as your records change and as vendors swap the model underneath. A 2022 peer-reviewed study in Scientific Reports observed temporal degradation in 91% of the cases it tested, using conventional machine learning models on 32 datasets of weather, airline, hospital and stock data. NIST's profile calls for "continuous monitoring of third-party GAI systems in deployment," meaning generative AI bought from outside. Listen for a vendor who commits in writing to tell you before the model underneath changes, and for a way to rerun your own known questions after each change.
-
What happens to our data and our work if we leave?
Leaving is part of buying, so ask what format your data comes back in and what the vendor deletes afterward. The ICO's guidance says a processor must, at the controller's choice, "delete or return" all the personal data it processed. In the EU, where the Data Act has applied since 12 September 2025, the European Commission's explainer says providers of software as a service must at a minimum "export data in a commonly used and machine-readable format," and switching charges end entirely from 12 January 2027. Listen for an export someone on your team has seen run, including the prompts and saved work your people built inside the tool.
-
What evidence stands behind the vendor's claims?
Claims about AI can run ahead of the testing behind them. In September 2024, the FTC alleged that a company selling a legal chatbot as "the world's first robot lawyer" had not tested whether its output matched a human lawyer's, and the company agreed to a proposed settlement. That was a consumer case, and the question carries to any purchase. Listen for what was measured, on what data, by whom, and how close that data sits to yours.
"What would this tool have to do on our own data for us to keep paying for it?"
The answers above add up to this one. Write it down before you sign, in plain words your team agrees on, and hold the renewal to it. Only your data and your people can answer it.
What are the signs an AI tool will work on your own data?
Good signs
- The vendor asks to run on your data before you ask.
- You hold a written list of every system the tool reads and every company that processes the data.
- The tool says when it cannot find an answer.
- Named people can say when in their week they will open it and what they will stop doing.
- Someone on your team has watched a full export of your data run.
Warning signs
- Every example in the demo comes from the vendor's sample accounts.
- Accuracy claims arrive without the data they were measured on.
- The contract lets the vendor change how it uses your data without telling you.
- Wrong answers in the trial read as confidently as right ones.
- The people meant to use it were never asked.
Case study Fictitious numbers
A PE-backed commercial pest control company has 26 sellers across five branches. Its VP of sales and its revenue operations lead are weighing an AI assistant that answers account questions before a seller walks a customer's site. In the vendor's demo, on the vendor's sample accounts, every answer was right.
Before signing, they connect a trial to their own CRM and put 60 questions to it from last year, each with an answer they already know. One asks which sites had a missed service in the quarter before the customer cancelled. The tool gets 38 right and 9 wrong. On the other 13, it says it cannot find the answer. That accounts for all 60.
- Right (38)
- Wrong, stated as certain (9)
- Said it could not find the answer (13)
The 13 refusals are easier to live with. A seller knows to look elsewhere.
The 38 get the least attention. All 9 wrong answers involve sites whose service history lives in the routing system the tool does not read, and each one reads as certain. The 13 refusals are easier to live with, because a seller knows to look elsewhere.
The tally is only part of the decision. The team asks the vendor to connect the routing data and run the same 60 again. They also ask two branch managers whether they would open the tool before a site walk in a normal week. One says yes. The other says he would keep asking his branch office, and that answer goes into the decision too.
FAQ
Can an AI vendor train its models on our CRM data?
It depends on the contract, so read it before you connect anything. Ask whether your data or your team's prompts train any model, the vendor's or a model provider's, and get the answer in writing. In February 2024, FTC staff warned that adopting more permissive data practices, such as AI training, through a "surreptitious, retroactive amendment" to terms may be unfair or deceptive. That post is staff guidance about promises to consumers. Ask how you will hear about any change.
Does ISO/IEC 42001 certification mean an AI tool is accurate?
No. ISO/IEC 42001:2023 is a management system standard. ISO says it specifies requirements for "establishing, implementing, maintaining, and continually improving" an AI management system inside an organization. A certificate tells you the vendor runs a governed process for AI. It does not tell you how the tool answers questions about your accounts. Count it as a good sign about the vendor, and still test the tool on your own data.
Should you trust an AI vendor's benchmark scores?
Use them as a starting point. NIST's Generative AI Profile notes that testing often stays "restricted to benchmark test datasets" that may not carry over to real-world conditions. It also asks organizations to document a benchmark's limits, including possible overlap between a model's training data and the test data. Ask what data produced the score and how close that data sits to your own records.
Is a free trial enough to evaluate an AI sales tool?
Only if it runs on your own data with the people who will use it. A trial in a sample workspace repeats the demo. Watch quality as closely as speed. In Pew's 2024 survey, four in ten workers who had used AI chatbots for work found them extremely or very helpful for doing things more quickly, and 29% said the same about the quality of their work. Check the answers along with the time saved.
Sources
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST, July 2024 checked 10 Oct 2026
- CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions, Huang et al., Salesforce AI Research, arXiv preprint, May 2025 (vendor research on synthetic data) checked 10 Oct 2026
- U.S. Workers Are More Worried Than Hopeful About Future AI Use in the Workplace, Pew Research Center, 25 Feb 2025 (survey fielded 7 to 13 Oct 2024) checked 10 Oct 2026
- Temporal quality degradation in AI models, Vela et al., Scientific Reports, July 2022 (peer-reviewed) checked 10 Oct 2026
- What needs to be included in the contract?, Information Commissioner's Office (UK GDPR guidance) checked 10 Oct 2026
- Data Act explained, European Commission, last updated 15 Dec 2025 checked 10 Oct 2026
- FTC Announces Crackdown on Deceptive AI Claims and Schemes, Federal Trade Commission, 25 Sep 2024 (allegations and proposed orders) checked 10 Oct 2026
- AI (and other) Companies: Quietly Changing Your Terms of Service Could Be Unfair or Deceptive, FTC staff, Technology Blog, 13 Feb 2024 checked 10 Oct 2026
- ISO/IEC 42001:2023, Artificial intelligence, Management system, ISO checked 10 Oct 2026