Why is the first version the cheap part of building AI?
The first version is the cheap part. In a 2015 peer-reviewed paper, a team of Google engineers wrote that developing and deploying machine learning systems "is relatively fast and cheap, but maintaining them over time is difficult and expensive." They wrote before today's large language models.
Analyst forecasts point the same way. In July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. In June 2025, it predicted that over 40% of agentic AI projects will be canceled by the end of 2027. Treat both as forecasts. Every cause Gartner names applies whether you build or buy.
For a revenue leader, upkeep competes for the hours you want spent on coaching and selling.
What questions decide whether to build AI in Claude or buy it?
Building fast is a real advantage, and a capable model makes a first version easy. These questions test the year after the demo. Ask them of your own team before you build and of any vendor before you buy.
-
What data does it need, and who maintains those connections?
A revenue tool is only as current as the CRM and service records it reads. A connection can break when someone renames a field or the company swaps a system. Listen for a named owner for each connection who knows when it last broke and what changed. A vague answer about "the integration" tells you nobody owns it.
-
How will we know its answers are right, week after week?
A demo proves one good afternoon. NIST's AI Risk Management Framework (January 2023) says AI systems "should be tested before their deployment and regularly while in operation." A 2024 study in the Harvard Data Science Review found that the behavior of two widely used language models changed substantially between March and June 2023, and its authors stress the need for continuous monitoring. Listen for a regular check against questions with known answers, and a named person who reviews the results, whether that person sits on your team or the vendor's.
-
Who owns it when the person who built it leaves?
A tool with one builder starts with a truck factor of one, meaning a single departure can stall it. A 2016 peer-reviewed study of 133 popular open-source projects on GitHub estimated that 65% had a truck factor of two or less. Those were public software projects, so read the figure as a caution about knowledge that sits with too few people. Listen for a second person who has already changed the tool and a manager who budgets time for its upkeep.
-
What does it cost in people's time, beyond the AI bill?
The model bill is the visible line. The larger cost sits in hours for cleaning data, checking outputs, fixing breaks and retesting when a model or system changes. Buying carries hours too, for setup and for managing the vendor. Listen for an estimate in hours per week with a name beside each hour, then weigh whether those hours would do more as coaching.
-
Is this something customers pay us for, or plumbing?
Customers pay you for the work and for how well you price and deliver it. The 2015 Google paper borrowed the word "plumbing" for much of the code that surrounds a machine learning model. Most internal tools play that role in a services business, and a renewal tracker is one of them. Build where your own data and know-how set you apart. Listen for someone who can name a customer who would notice if the tool disappeared.
-
What can it see and change, and who has checked?
A tool that reads customer email and writes to the CRM can be steered by what it reads. The OWASP Top 10 for LLM Applications (2025) lists prompt injection first and notes that indirect injections arrive through external sources such as websites or files. Building keeps these choices in your hands, and NIST's framework notes that technology acquired from third parties "may be complex or opaque," with risk tolerances that may not match yours. Listen for who reviewed its access and which changes still need a person to approve them.
"Who maintains this a year from now?"
Before you build or buy anything this quarter, write down who will maintain it a year from now. If no name comes to mind, wait.
What are the signs a home-built AI tool will last?
Good signs
- Each data connection has a named owner who knows when it last broke.
- Someone other than the builder has changed the tool and could run it tomorrow.
- The team checks answers against known results on a regular rhythm and can show the last miss and its fix.
- The cost case counts people's hours alongside the model bill.
- The build sits close to how you win work and runs on data only you hold.
Warning signs
- The only documentation is the builder's memory.
- Nobody can say when the answers were last checked.
- Managers stopped questioning the output months ago.
- Several teams have built their own versions of the same report.
- A vendor cannot explain what happens to your data or how it handles a model change.
Case study Fictitious numbers
A PE-backed commercial fire protection company has 18 sellers across three branches. Its sales operations lead spends two weeks building a renewal tracker on a general-purpose model. The model bill runs about $100 a month, or $1,200 a year. Upkeep takes about 5 hours a week of her time, spent fixing data exports and spot-checking answers. At a loaded cost of $70 an hour, that is $350 a week, or $18,200 over 52 weeks. The full annual cost is about $19,400, and the model bill is about 6% of it.
- Model bill
- People's hours: fixing data exports and spot-checking answers
- Customers would not notice it, so it is plumbing
- The data feeds and checks depend on one person
- A bought tracker came in lower, setup and vendor hours included
- Prices bids from the company's own job history
- Shapes how the company wins work
- The estimating manager owns it, two hours a week
Both choices pass the same test: who maintains it a year from now.
The data feeds and the checks depend on her alone. Customers pay for inspections and service agreements, and none of them would notice the tracker. That makes it plumbing. The team prices a bought tracker the same way, subscription plus setup and vendor hours, and it comes in lower. Renewal tracking moves to the bought tool, and her hours go back to the branches.
The same company keeps building in one place. Its estimators use a model-based helper to price bids from the company's own job history. Bid pricing shapes how the company wins work, and the estimating manager has two hours a week set aside to own it. Both choices pass the same test.
FAQ
Is it cheaper to build AI sales tools in-house?
Sometimes. The model bill for a small internal tool can be modest. The bigger cost is people's hours, from data upkeep to the security review and the retest after a model change. Price those hours at loaded cost for a full year before you compare. Buying has hours of its own for setup and vendor management, so count both sides the same way.
Do internal AI builds fail more often than bought tools?
The public evidence is thin, and the causes Gartner names for abandoned AI projects can sink either path. Buying carries a risk of its own. In June 2025, Gartner warned of "agent washing," rebranding products such as chatbots without substantial agentic capabilities, and estimated that only about 130 of the thousands of agentic AI vendors are real. Hold a build and a vendor to the same questions, and staff whichever one you choose.
When does it make sense to build your own AI tool?
Build when the tool shapes how you win work and someone owns it with protected time. Build when the stakes are low and the main user is you, such as a first draft or a one-off analysis. A small build also teaches you what to ask vendors. Even the 2015 Google paper noted that when glue code piles up around a generic package, a clean in-house build may cost less.
What happens when the AI model behind your tool is retired?
Someone has to move the tool and retest it. Anthropic's documentation says it gives at least 60 days' notice before retiring a publicly released model, and requests to a retired model fail. Its deprecation table shows a model snapshot dated February 2025 retired in February 2026. A vendor building on any model faces the same change, so ask how they test a replacement before the switch.
Sources
- Hidden Technical Debt in Machine Learning Systems, Sculley et al., NeurIPS 2015 (peer-reviewed) checked 9 Oct 2026
- Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025 (analyst prediction, 29 Jul 2024) checked 9 Oct 2026
- Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (analyst prediction, 25 Jun 2025) checked 9 Oct 2026
- NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0), January 2023 checked 9 Oct 2026
- How Is ChatGPT's Behavior Changing Over Time? Chen, Zaharia and Zou, Harvard Data Science Review, 2024 checked 9 Oct 2026
- A Novel Approach for Estimating Truck Factors, Avelino et al., IEEE ICPC 2016 (peer-reviewed) checked 9 Oct 2026
- LLM01:2025 Prompt Injection, OWASP Top 10 for LLM Applications checked 9 Oct 2026
- Model deprecations, Anthropic Claude Platform Docs checked 9 Oct 2026