The demo always goes well. The dashboard loads instantly, the model flags the right patient, the reference customer is enthusiastic, and the price looks manageable against the savings slide. Eighteen months later, the same tool is live in three wards instead of thirty, the clinicians who championed it have moved on, and the renewal lands on a procurement desk with nobody able to say whether it worked.
Hospitals are not bad at evaluating technology. They are bad at evaluating the parts of a digital purchase that don’t show up in a demo. And the standard procurement checklist, focused on features, security certificates and price, is built to answer the questions vendors are best prepared for.
Here are the questions that tend to get asked after the contract is signed, when they should be asked before.
“Show me your performance on data that looks like ours”
Vendors usually present accuracy figures from their development data or their best-performing site. That number tells you little about your hospital.
The clearest warning is the Epic Sepsis Model. When researchers at Michigan Medicine validated it on their own patients and published the results in JAMA Internal Medicine in 2021, the model missed 67% of sepsis cases while alerting on about 18% of all hospitalisations. It was embedded in one of the most widely used health record systems in the United States, which made it easy for any single hospital to assume someone else had checked.
The question to ask is not “how accurate is it?” but “will you run it silently on twelve months of our historical data before go-live, and will you agree in the contract on what happens if it underperforms?” A vendor confident in its product should say yes. Hesitation is information.
“What happens to the data, including the parts nobody thinks of as data?”
Most hospitals ask where patient records will be stored. Fewer ask what the product sends elsewhere as a by-product of normal use.
The US Federal Trade Commission’s cases against GoodRx and BetterHelp in 2023 both involved health information flowing to advertising platforms through standard tracking tools. GoodRx paid a $1.5 million civil penalty; BetterHelp agreed to pay $7.8 million. Neither was a hospital vendor, but the mechanism, third-party code embedded in an app or web page, is the same one that sits inside many patient-facing products that hospitals license and brand as their own.
Ask for a list of every third-party SDK and tracking script in any patient-facing component, and make changes to that list require notice.
“Who in our organisation has to change what they do?”
This is the question that most often decides success, and it is almost never in the tender documents.
A digital tool rarely replaces work cleanly. A remote monitoring platform needs someone to watch the alerts. A discharge prediction tool needs someone to act on it before noon. A patient app needs someone to onboard patients and answer when they get stuck. If the answer to “who does this?” is “the existing team, in their spare time,” the tool will be used by enthusiasts and ignored by everyone else.
The experience of prescription digital therapeutics makes the point. When Medicare introduced codes for digital mental health treatment devices in its 2025 physician fee schedule, it paid for the device supply and, separately, for clinician time spent managing its use. Even the payer recognised that the software doesn’t run itself.
“What will this cost in year three?”
Digital health pricing often looks cheap in the first year: a pilot rate, a subsidised implementation, a small number of licences. The real cost shows up later, in per-patient fees that scale with success, integration work with the health record, and internal staff time.
Ask the vendor to model total cost at full deployment, including your own staff, and ask what the price becomes if the product works and usage grows. Then ask how easily you can leave: can you export your data in a usable format, and what happens to patients mid-programme if the contract ends?
“Will you still be here?”
It feels impolite to ask, but it matters. In April 2023, Pear Therapeutics, the first company to receive FDA authorisation for a prescription digital therapeutic, filed for bankruptcy. Payers that had signed deals with it, including Oklahoma’s Medicaid programme, were left to reassess.
In Germany, the DiGA directory offers a reminder that products can also disappear for evidential reasons: by mid-2025, eleven apps had been removed, six because they could not demonstrate a positive healthcare effect.
A hospital cannot audit a startup’s runway. It can ask for escrow of critical software, data portability, and a transition plan written into the contract.
The incentive underneath
The awkward truth is that the people who buy digital health are rarely the people who use it, and neither group usually owns the outcome it was bought to improve. The IT team owns integration. Procurement owns price. Clinical leaders own adoption. Finance owns the savings case. Nobody owns the question “did this make patients better off?”
Vendors understand this and sell to each group separately. A hospital that wants better results should do the opposite: name one accountable owner for each purchase, give them a measure agreed before signing, and make renewal depend on it.
Why good hospitals still make these mistakes
It is tempting to blame poor procurement on inexperience. The more accurate explanation is structural.
Most hospital purchasing processes were designed for equipment and supplies. When you buy an infusion pump, the product is the thing you get: it arrives, it works as specified, and it keeps doing so. The main questions are price, reliability and service. Procurement teams are good at those.
Digital health products are different in three ways. Their performance depends on your data, which differs from the vendor’s. Their value depends on changes in how people work, which the vendor cannot deliver. And their cost depends on how much they are used, which is often unknown at the time of purchase. A procurement process built for pumps asks the wrong questions of software, and the vendor has no reason to point that out.
There is also a timing problem. Clinical champions usually find a product first, often at a conference or through a peer. By the time procurement is involved, a decision has effectively been made and the process becomes a matter of paperwork. The hard questions get asked late, if at all, because asking them feels like obstruction.
A worked example: the remote monitoring contract
Consider a hospital that buys a remote monitoring platform for patients with heart failure after discharge. The business case rests on a reduction in readmissions seen at a reference site.
In month one, the platform goes live. Patients receive a tablet, a blood pressure cuff and a scale. Readings flow to a dashboard.
In month two, the dashboard is producing dozens of alerts a day: small weight changes, missed readings, borderline blood pressures. The heart failure nurses, who were expected to monitor it alongside their clinic work, can’t keep up. Alerts are triaged by exception, then glanced at, then batched to the end of the day.
In month six, the evaluation shows no reduction in readmissions. The reference site, it turns out, had funded two dedicated nurses to monitor the platform, and its patients were younger and more digitally confident. The product was not the difference. The staffing was.
Every step of this story would have been visible if someone had asked, before signing: what staffing did the reference site use, and what alert volume should we expect per hundred patients? These are simple questions. They are rarely asked because they make the business case look worse.
The strongest counterargument
The case against a more demanding procurement process is that it slows innovation to a crawl.
Health systems are already notoriously slow to adopt technology. Asking vendors for local validation, detailed staffing models, SDK disclosures, escrow arrangements and outcome-based contracts will exclude smaller companies that can’t afford long sales cycles. The result could be a market dominated by large incumbents, which is exactly the problem many hospitals already have with their record systems. Some promising tools will never be tried.
This deserves weight. Excessive process is a real cost, and hospitals that ask for everything will get products from those who can afford the paperwork, not necessarily the best.
The answer is proportionality, not abandonment. A low-risk scheduling tool doesn’t need the same scrutiny as a clinical prediction model or a patient-facing app that handles sensitive data. Hospitals can create a fast track for low-risk, low-cost pilots with a clear evaluation plan, and reserve the heavier questions for products that touch clinical decisions, patient data at scale or large budgets. The mistake is not asking hard questions. It is asking them of everything or of nothing.
Second-order effects
Vendor lock-in accumulates quietly. Each product that stores data in a proprietary format or integrates deeply with one record system makes the next change harder. Over years, a hospital can find its options narrowed not by any single bad decision but by many reasonable ones.
Pilot fatigue sets in. Staff who have been through several digital pilots that fizzled become sceptical of the next one, even if it is good. Poorly chosen tools raise the cost of adopting good ones.
Evidence stays thin. When hospitals don’t measure outcomes after purchase, nobody learns which products work. Vendors keep citing the same reference site, and the next buyer faces the same uncertainty.
A short pre-signing checklist
Pulled together, the questions above fit on one page, and that is roughly how short they should be:
- Has the tool been validated on our own historical data, and what happens contractually if it underperforms?
- Which third-party code runs in any patient-facing part of the product?
- Who in our organisation does new work because of this tool, and is that time funded?
- What is the total cost at full deployment in year three, including our own staff?
- How do we get our data out, and what happens to patients if the vendor fails or we leave?
- Who owns the outcome, what is the measure, and when is the review?
A vendor who can answer all six clearly is rare. That is precisely why the questions are worth asking.
What this means for decision-makers
For hospital boards, the practical step is to require that every significant digital purchase has a named clinical owner, a pre-agreed outcome measure and a review date, and that renewal is conditional on the review.
For chief information and chief medical officers, it is to build a shared evaluation capability: people who can run a silent validation on local data, estimate workload, and look at what an app sends over the network.
For clinical champions, it is to be the person who asks the awkward questions, not the one who avoids them to get the product through. Your credibility depends on the tool working, not on it being bought.
For ministries and national agencies, it is to reduce the duplication of effort by sharing evaluations across hospitals. Every hospital independently validating the same tool is wasteful. A national or regional evaluation hub, publishing results, would help buyers and honest vendors alike.
What to watch next
Watch for hospitals and health systems that publish post-implementation evaluations, including failures. Watch for contract terms that tie payment to outcomes. And watch for regulators taking a closer look at clinical AI tools embedded in record systems, where the Epic sepsis experience showed that wide deployment is no substitute for local evidence.
The takeaway
Most failed digital health purchases were not bad products bought by careless people. They were reasonable products bought without asking who would use them, how they would be validated locally, and what would happen when things went wrong.
Ask those questions before the demo, not after the renewal notice.
What’s the one question you wish your organisation had asked before signing a digital health contract? Share it below.
