Most AI pilots never reach production, and the reason is rarely the model. A practical guide to scoping an AI engagement: which use cases pay off, how running cost behaves, how to prove the system works, and where the EU AI Act now stands.
By 2026 almost every mid-sized company has run an AI pilot. A large share of them never reached production. The interesting part is that the model is almost never the reason — capability has not been the binding constraint for a while. The reasons are mundane, repeatable, and mostly avoidable in the first six weeks.
The four ways AI projects actually fail
- The pilot was demonstrated on curated inputs and collapsed on the real ones. Every organisation's data is messier than the sample used in the demo.
- Nobody agreed what accuracy would count as success, so when the number arrived there was no way to say whether it had passed.
- It was never integrated into the workflow. Using it meant opening a separate tool, so the people it was built for quietly stopped.
- Running cost was never modelled and only became visible at production volume, at which point the business case evaporated.
All four are cheaper to catch in a two-week feasibility stage than in a six-month build. That is the main argument for consulting as a distinct phase rather than going straight to a development contract.
PerceptiaAI
Ready to transform your business with AI?
What an AI consulting engagement should contain
A useful engagement has four stages, each ending in something you can hold and act on independently. If a proposal has only the first stage, you are buying a strategy document; if it has only the third, you are buying a build against unvalidated assumptions.
| Stage | Duration | What you should get |
|---|---|---|
| Opportunity assessment | 1-2 weeks | A ranked shortlist with estimated value, estimated build cost and feasibility per use case — including the ones recommended against |
| Feasibility and data readiness | 2-4 weeks | A prototype tested against your real data, and an honest read on whether accuracy clears the threshold you set in advance |
| Production build | 6-16 weeks | The system running inside your workflow, with monitoring, an evaluation suite and a documented fallback |
| Evaluation and improvement | Ongoing | A regression suite running on every change, live quality metrics, and a regular review of where it is failing |
Which use cases reliably pay off
The pattern that works is high volume, high repetition, and a human decision that follows describable rules. The pattern that does not is anything where being wrong occasionally is unacceptable and no one is reviewing the output.
| Usually worth doing | Usually not worth doing |
|---|---|
| Document extraction, classification and summarisation at volume | Anything where a 5% error rate is unacceptable and nothing is reviewed |
| Triage and routing where a human decides today along patterns you can describe | Problems a database query or a rules engine would solve more cheaply and predictably |
| Internal knowledge retrieval across scattered docs, tickets and past work | Use cases with no ground truth — if you cannot define a correct answer, you cannot evaluate it |
| First-draft generation reviewed by a person before it leaves the building | Workflows nobody has documented, where the AI project is really process-mapping avoidance |
| Structured extraction from emails, PDFs, forms and transcripts | Low-volume tasks where the engineering and running cost will never be repaid |
Buy, build, retrieve or fine-tune
Four architectural routes, with very different cost and control profiles. Teams tend to reach for the last one first and need it least.
| Approach | Choose when | Watch out for |
|---|---|---|
| Buy a product | Common problem, mature product, your data is not the differentiator | Per-seat pricing at scale, data leaving your control, integration work exceeding the licence |
| Build on a hosted model API | Problem specific to you, volume justifies engineering, needs to sit inside your systems | Inference cost at real volume, rate limits, model versions changing behaviour under you |
| Retrieval over your own content | The value is in your documents and records rather than general reasoning | Retrieval quality decides the outcome, not model choice. Enforce permissions at retrieval time, never in the prompt |
| Fine-tune or train | You need consistent structure or a narrow domain style and have thousands of labelled examples | Prompting plus retrieval solves most of what fine-tuning gets proposed for, at a fraction of the cost |
Six questions that decide whether this is feasible at all
- Can you point to where the data lives — not which system in principle, but which table, bucket or folder, and who grants access?
- Is there a correct answer to compare against? Historical decisions, resolved tickets, approved documents.
- How much of the data is unusable? Duplicates, scans with no text layer, records abandoned years ago. This is always larger than expected and usually survivable.
- What is in it that must not leave? Personal data, health records, financial identifiers. This determines the architecture, so it is a first-week question.
- Who owns it, and can it be used this way? Contractual restrictions on client data, and consent basis for personal data.
- How does it change? A system built on a monthly export behaves differently from one built on a live feed.
If four or more have clear answers, a feasibility stage is worth running. If fewer, the first project is a data project, not an AI project.
How AI running cost behaves, and why it surprises people
Traditional software costs scale with users. AI systems scale with usage, which is a different curve and a different budgeting conversation. The main drivers:
- Inference, charged per token or per request. The gap between a naive implementation and a considered one is routinely five to ten times on identical workloads.
- Retrieval infrastructure: vector storage, re-embedding when documents change, and the pipeline keeping the index current.
- Evaluation. Building the test set is a real cost, and skipping it is why teams cannot tell whether last week's change helped.
- Human review. Where output is checked, reviewer time is part of the running cost and belongs in the business case from day one.
- Maintenance. Providers deprecate models, change defaults and adjust pricing. Launch is not the end of the spend.
Caching, routing simpler requests to cheaper models and limiting context are architectural decisions with a direct monthly cost attached. They belong in the design, not in a later optimisation pass.
Proving it works: evaluation is the whole difference
The distinction between an AI pilot and an AI system is an evaluation harness. That means a test set built from your own examples, run automatically on every change, plus live quality metrics in production. It also means designing for being wrong: a confidence threshold below which output goes to a human, an audit trail of what the system saw and produced, and a fallback that keeps the workflow moving when the model or the provider is unavailable.
If a vendor cannot tell you how accuracy will be measured before the build starts, they are not planning to measure it.
Where the EU AI Act stands as of August 2026
If your system is used in the EU, the AI Act applies regardless of where your company is based, and the timeline moved materially this year. The Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 27 July 2026 — days before the original high-risk deadline — and deferred several obligations.
| Obligation | Applies from | Status |
|---|---|---|
| Prohibited practices (Article 5) | 2 February 2025 | In force, unchanged |
| General-purpose AI provider obligations | 2 August 2025 | In force, unchanged |
| Transparency and AI-content labelling (Article 50) | 2 August 2026 | In force, unchanged |
| High-risk, standalone systems (Annex III) | 2 December 2027 | Deferred from 2 August 2026 |
| High-risk, AI embedded in regulated products (Annex I) | 2 August 2028 | Deferred |
For most business automation the practical consequence is narrower than the headlines suggest: the majority of internal document and workflow systems are not high-risk. The transparency duties, however, frequently do apply, and they are already live. Either way, the technical requirements are the same things good engineering asks for anyway — audit trails, disclosure where users are interacting with an AI system, human review where output carries weight, and documented data handling. This is a summary of the regulatory position, not legal advice; classification of a specific system is a question for counsel.
What to ask an AI consultant before signing
- Which of our use cases would you tell us not to build, and why?
- How will accuracy be measured, and what threshold counts as success?
- What will this cost to run per month at our expected volume?
- What happens, concretely, when the model gets something wrong?
- Will our data be excluded from provider training, and how is that documented?
- What do we own at the end of each stage if we stop there?
Take Your Next Step
Whether you're looking to integrate AI into your workflow or just want to see more of our industry insights, we're here to help you lead the market.
Written by
PerceptiaAI Team