The AI vendor market is now full of companies that wrap a general-purpose model behind a thin user interface, give it a name, raise a round, and start selling. That is not inherently a bad thing. Some of the most useful tools we have ever recommended are wrappers in the literal sense. The team behind them has done excellent work figuring out a specific workflow, the prompts behind it, the data plumbing, and the change management. The wrapper is the product.
Other wrappers are not that. They are a landing page, a Stripe link, and a chatbot. The market has gotten harder to read, because the good ones and the bad ones look identical from the outside. The website does not tell you which is which. The demo does not either.
Why the demo stopped working
A demo is a tour of a vendor's polished example data, walked through by someone who has run this same demo two hundred times. Every input has been rehearsed. Every edge case has been routed around. Every failure mode has been quietly hidden from the script. You will see the system at its best, and only at its best.
What you actually need to know is how the system behaves on your data, in your workflow, when something it has never seen before lands in the queue. The demo does not tell you that. The vendor's reference customers, hand-picked and pre-briefed, will not tell you that either. The five questions below are designed to find out.
The five questions we ask every AI vendor
1. What happens to our data?
Specifically: where does the data go, how long is it retained, who has access to it, is it used to train any model, and what happens to it if we cancel. The answer should be on the vendor's website. If it is not, that is a signal. If it is on the website but the salesperson cannot explain it from memory, that is a bigger signal.
Pay particular attention to whether your data is used to improve the model. Some vendors treat your data as theirs the moment it crosses their API. Others treat it as yours, with explicit opt-in for any reuse. The difference matters more than the price.
2. What is the model strategy?
Do they train their own model, or do they call out to one of the major providers? Both can be fine. The wrong answer is "we are flexible." A serious vendor has made a choice, has a reason for it, and can describe what happens to your workflow when the underlying model they depend on changes its pricing, deprecates a feature, or quietly shifts its behavior.
If the vendor is a wrapper around a major model, ask what they are doing on top of the model that is worth paying for. Sometimes the answer is excellent. Prompt engineering done well is a real skill, data integration is harder, change management is the hardest. Sometimes the answer is nothing. The question separates the two.
3. Who has actually deployed this in production?
Ask for three customers running the system in production today, not in pilot. The distinction matters. A pilot is a sandbox. Production is the thing your team will live inside. Plenty of AI vendors have impressive pilot counts and almost no live production deployments, because the gap between the two is where most AI projects die.
When the references come back, the question to ask is not "do you like it." Everyone likes the vendor they just paid. The question is "what does it look like when it breaks, and how often does that happen, and what happens next." The answer to that question tells you whether you are buying a finished product or a research project with a logo.
4. What is the price doing in eighteen months?
AI pricing is unstable. The underlying model costs are dropping. Some vendors are passing the savings on. Others are quietly increasing the price as they figure out what the market will bear. Both are legitimate business decisions, but you should know which one you are dealing with before you sign.
Ask whether your contract locks the price. Ask what a renewal usually looks like. Ask what happens to your bill if you double the volume. The answers will not all be in writing. The conversation is the signal.
5. What does support look like at 2am?
AI systems fail in interesting ways. The model returns nonsense. The integration drops messages. An upstream provider has an outage. A prompt that worked last week stops working this week. None of these are catastrophes. All of them require a real human to answer the phone.
Ask how support is structured. Ask the size of the engineering team behind the product. Ask what the median response time on a serious ticket actually is. A vendor with a thin team and a chatbot for support is selling you a tool that will be unsupported when the second outage hits, regardless of what the sales team promises now.
What separates a real AI company from a thin wrapper
It is not whether they trained their own model. Most serious AI companies do not, and that is now a sound business decision. The thing that separates the real ones is how much of the work they have done on top of the model. The integrations they have built. The edge cases they have learned. The reliability work they have invested in. The customer support muscle. The vertical knowledge.
A well-built wrapper is more useful than a poorly-built proprietary model, every time. The mistake to avoid is assuming that one category is automatically better than the other. The category does not matter. The execution does.
Where this fits into our work
Vendor diligence is one of the things we end up doing inside almost every engagement, not because it is glamorous, but because the cost of picking wrong is high and the people running the business do not have the time to run five separate vendor evaluations a quarter. We do that work between exec meetings, and the output is usually a short memo with a recommendation and a reason.
If you are running this process yourself and want a sanity check on a specific vendor, the offer is open. We have no commercial relationship with any of the vendors in the market, which is deliberate. The conversation is short, the recommendation is honest, and sometimes it is "this one is fine, sign and move on." That is the version of vendor work worth doing.
