An AI proof of concept checklist: before, during and at the end

An AI proof of concept has one job: letting someone with a budget decide what happens next. If, after weeks of work, leadership receives a polished technical report and still can't tell whether to scale, adjust or drop the idea, the PoC failed, even if the model got everything it tried right.

This kind of failure is common, and it shows up in the research. In July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, pointing to poor data quality, inadequate risk controls, escalating costs and unclear business value. A year later, the same firm estimated that over 40% of agentic AI projects will be canceled by the end of 2027, for cost, unclear value and weak risk controls.

RAND Corporation's 2024 report on the root causes of AI project failure interviewed 65 experienced data scientists and engineers. The most cited cause was misunderstanding, or miscommunicating, what problem the AI was supposed to solve. That produces models optimized for the wrong metric, or models that don't fit the real workflow.

Almost none of these causes are technical. Bad data, rising cost, value nobody defined, risk nobody measured: all of it can be found inside a well-designed PoC, for a fraction of what it costs to find in production. The checklist below is how we make a PoC look for those answers on purpose.

Before day one

Most of a PoC's outcome is decided before anyone writes code.

1. The decision question fits in one sentence. Not "test AI in customer service" but "should we fund automating tier-2 ticket triage next half?". If the question doesn't fit in a sentence, the scope is still open.
2. There is a numeric success criterion, with thresholds. Accuracy, time saved per case, cost per transaction. Thresholds are agreed up front: above one value we scale, in a middle band we adjust, below it we drop. A criterion interpreted after the result always favors continuing.
3. There is a baseline. How long the process takes today, the human error rate, what a simple rule could do. Google's Rules of Machine Learning open with exactly this: don't be afraid to launch a product without machine learning, and start with heuristics. If a ten-line rule solves a good part of the problem, the AI has to beat the rule, not zero.
4. Access to real data is approved. In a large company, opening a production dataset goes through security, legal and often the privacy officer. If that isn't settled at kickoff, the first weeks disappear waiting.
5. The decision owner has a name. One person, with budget, who will read the result and decide. Not a generic committee.
6. Timeline and budget are fixed. In the Cyborg Forge we work in 3 to 6 weeks. An open timeline turns a PoC into a disguised project, without a project's controls.

During the build

• Test with real data, including the ugly cases. A hand-picked sample produces a good demo and a bad decision. Pull out the hard cases (crooked scans, customers who send voice notes, incomplete records) and measure on them.
• Measure unit cost from the start. In generative AI, cost per model call, per document or per conversation decides whether the idea pays off at scale. Rising cost is one of the abandonment reasons Gartner lists, and it's easy to measure in a PoC.
• Log the errors, not just the error rate. A high accuracy number doesn't tell you whether the remaining errors are cheap or whether one of them triggers a fine. A spreadsheet with a few dozen real errors, ranked by severity, is worth more to leadership than any chart.
• Write down what was left out. ERP integration, peak volume, languages, user profiles. Everything the PoC didn't test becomes an explicit risk item in the report.
• Look at risk alongside results. Personal data in the prompt, answers that could be read as advice, automated decisions about customers. The NIST AI Risk Management Framework is a free, practical reference for organizing that list.

At delivery

The final deliverable has to be readable by the decision-maker in fifteen minutes. At Cortex IQ we call it the Decision Package, and it always has the same parts:

1. The original decision question, copied from the kickoff document.
2. The measured result against the agreed criterion and against the baseline.
3. The business impact estimate, with assumptions written out: monthly volume, cost per hour, expected adoption. Hidden assumptions are the most common way to inflate a business case.
4. The estimated cost of running it in production: infrastructure, model calls, monitoring, people.
5. Known limitations and what the PoC did not test.
6. The recommendation: scale, adjust or drop, and why.
7. If the recommendation is to scale, what production still needs: integrations, an operational owner, a monitoring plan.

A worked example, with hypothetical numbers

Take a distributor that receives purchase orders by email, as PDFs, in layouts from dozens of different customers. Today two assistants key those orders into the ERP.

The decision question: "should we automate reading PDF orders this year?". The criterion: correctly extract the required fields on at least 95% of orders, at a cost per order below today's manual entry. The baseline: measure, for one week, how long each order takes and how many come back because of typing errors.

The PoC runs on three months of real orders, scanned ones included. Suppose the result is 97% accuracy on native PDFs and 81% on scans, which are a fifth of the volume. A poorly designed PoC would report "94% average accuracy" and leave leadership guessing. A well-designed one recommends scaling native PDFs now, keeping scans under human review, and revisiting scans in six months. That is a decision someone can make in one meeting.

The mistakes we see most

Criteria set afterward. Once results are in, there's always a reason why 80% is good enough. If nobody wrote down beforehand that the floor was 90%, the discussion turns into opinion.

The demo standing in for evidence. A live demo wins the room and proves nothing about real volume. Use the demo to show how it works and the error log to show whether it works.

The "adjust" that never ends. Adjusting is a valid recommendation when it has a deadline and a new criterion. A second round of adjustments with no date is a PoC nobody had the nerve to close.

Dropping treated as failure. A PoC that concludes "not worth it" in five weeks saved the budget of an entire project. If the company culture punishes that outcome, teams stop recommending it, and PoCs become projects nobody can switch off.

Where to start

Before approving the next proof of concept, ask the team for one page with three things: the decision question in one sentence, the numeric criterion with scale, adjust and drop thresholds, and the name of whoever will decide. If that page isn't ready within a week, the problem isn't technical and no PoC will fix it.

If you want a ready-made format, the Cyborg Forge runs this process on AWS or Google Cloud and ends with the Decision Package described above. For larger data and modeling initiatives, see our Data and AI work.