When a machine learning model doesn't make it to production, the reason is almost never accuracy. The model works in the notebook, the metric looks good, the presentation goes well, and six months later nobody uses it. What's missing is everything that has to exist around the model for a prediction to turn into an action inside the company.
What the numbers say
A Gartner survey released in August 2022 found that, on average, 54% of AI projects make it from pilot to production, roughly the same level as the 2019 edition of the survey, which had reported 53%. Close to half fall short, and the number barely moved in three years.
RAND Corporation's 2024 report on the root causes of AI project failure cites estimates that more than 80% of these projects fail, twice the rate of IT projects without AI. Based on interviews with 65 data scientists and engineers, the authors identify five leading causes: misunderstanding the problem to be solved, lacking the necessary data, chasing the latest technology instead of the problem, underinvesting in infrastructure, and applying AI to tasks it can't yet handle well.
There's an older explanation for the same pattern. In 2015, Google engineers published Hidden Technical Debt in Machine Learning Systems at NeurIPS, showing that in real systems the model code is a small part of the whole. Around it sit data collection and validation, feature extraction, configuration, serving infrastructure and monitoring. Google Cloud's MLOps guide makes the same point. If you budget only for the model, you've budgeted for the smallest part of the work.
The real reasons, in the order they show up
Nobody designed the action
A churn model with good accuracy doesn't reduce churn. The retention team calling the right customer with the right offer does. If that team can make 50 calls a week, what matters is whether the top 50 on the list are really the ones about to cancel, not the average accuracy across the whole customer base.
That changes the metric (precision at the top of the ranking, not an overall metric like AUC), the delivery format (a weekly list in the CRM, not an API) and who needs to be in the room from day one. It's RAND's first root cause in practical form: a model optimized for the wrong metric, outside the real workflow.
Production data isn't training data
The model is trained on a clean history exported in one go. In production, data arrives incomplete, late, with a field whose format changed after an ERP update. Google's guide names one symptom of this training-serving skew: a feature computed one way in training and another way in production.
A typical example is a "days since last purchase" feature. In the notebook it's computed over the full dataset, sometimes accidentally using information from after the prediction date. In production it's computed on the day, with whatever data is available at that moment. The model that looked excellent in testing loses performance without anything visibly breaking.
The fix is engineering: the same function computes the feature in training and in production, inputs are validated before they reach the model, and the distribution of key variables is monitored. When the data profile shifts (a new sales channel, a price change, a seasonality the history didn't have), someone gets an alert.
Nobody owns it after launch
A model in production is a system that degrades over time, because the world it describes changes. It needs someone who watches the monitoring, decides when to retrain and responds when the business says "the model is acting weird". If the data scientist who built it has moved to another project and nobody inherited responsibility, the model keeps running until someone switches it off out of distrust.
The handoff from notebook to engineering was never planned
Google's guide describes MLOps maturity level 0: everything manual, a data scientist hands a model file to an engineering team to deploy, no pipeline, no CI/CD, infrequent releases. It's where many companies are, and where many models die, in the queue of a team that didn't help build it and has no priority for it.
The offline metric doesn't connect to the business metric
A recommendation model can improve its offline metric and not move revenue at all. Without a controlled test in production (an A/B test, or at least a period in shadow mode, running in parallel without affecting decisions), nobody can claim the model created value. Without that evidence, it won't survive the first budget review.
Before training, start without machine learning
Google's Rules of Machine Learning, written by Martin Zinkevich, open with a rule that would prevent many stalled projects: don't be afraid to launch a product without machine learning. A simple heuristic, such as "customers with no purchase in 60 days and an open complaint", put into production first solves three problems at once. It forces you to design the action, it builds the data pipeline the model will use, and it sets the baseline the model has to beat. If the model can't beat the rule, you just saved the project.
A readiness checklist
The ML Test Score, another paper by Google researchers, proposes 28 tests and monitoring checks to gauge whether an ML system is ready for production. For a company getting started, a reduced version already filters out a good share of the problems:
- The decision the model supports is written down, including who acts and at what capacity (how many cases per day or week).
- The evaluation metric reflects that capacity, and there's a non-ML baseline to compare against.
- The code that computes features is the same in training and in production.
- Input data is validated before prediction: nulls, out-of-range values, new categories.
- The model is versioned, and you can roll back to the previous version in minutes.
- Input and prediction distributions are monitored, with alerts.
- There's a written retraining criterion: a schedule, a performance drop or a data shift.
- The model has a named owner after launch.
- Impact was measured in production against a control group, or there's a plan to measure it.
MLOps without overdoing it
None of this requires a sophisticated platform for the first model. For many mid-sized companies, it's enough to have a scheduled job that runs batch predictions every night, writes the results to a table the CRM or ERP already reads, records which model version was used and feeds a weekly monitoring dashboard. Full pipeline and retraining automation, what Google calls levels 1 and 2, makes sense once several models are in production or when data changes fast enough to require frequent retraining.
The opposite mistake happens too: building a full MLOps platform before the first model creates any value. Infrastructure without a use case is fixed cost waiting for a justification.
Where to start
Take the model sitting in a notebook and answer in writing: who acts on the prediction, how often, inside which system, and who will be accountable for it six months from now. If any answer is blank, that's where the project is stuck, and no accuracy improvement will unstick it.
If you want help taking a model into operations, with pipeline, monitoring and ownership in place, that's the work of our Machine learning service. If the open question is still whether the use case pays off, a PoC Factory engagement answers it in a few weeks, with real data and criteria agreed up front.