Published
Updated
This guide shows how to design an AI pilot that leads to a decision, and then what the rollout adds. It assumes you have already chosen the use case: the guide prioritising AI use cases covers that choice.
Key points
A pilot exists to decide: stop, fix or extend.
Before you start, write down the baseline, the success criterion and the stop criterion.
Scaling up adds costs that the pilot does not show: recurring budget, support, training, maintenance.
What is an AI pilot for?
An AI pilot exists to decide whether to extend, fix or stop a use, based on measurements taken in real work. It differs from a technical demonstration: the explainer POC, pilot and scale-up (in French) sets out this distinction.
Write down three things before day one:
the baseline: the current time, quality and cost of the task, measured over a few weeks;
the success criterion: the quantified result that justifies extending;
the stop criterion: the result that justifies abandoning, and the date on which you will check it.
Without a baseline, you can only compare impressions. The explainer AI project ROI (in French) details these notions.
Why do so many AI projects fail?
The failures RAND describes stem mainly from the problem as defined, the data and the organisation, more than from the model. The RAND Corporation report published in August 2024 draws on 65 interviews with experienced data scientists and engineers. It identifies five main causes:
leadership misunderstands or poorly communicates the problem to solve;
the organisation lacks the data to train an effective model;
the team focuses on the latest technology rather than on users’ problems;
the infrastructure cannot manage the data or deploy;
the problem exceeds what AI can do.
The authors flag a bias: their sample consists mostly of engineers, who are more inclined to point to leadership failures. A well-designed pilot tests causes 1, 2 and 4 before committing to the rollout.
How do you choose the scope of an AI pilot?
Choose a scope that represents the intended rollout: several teams, varied profiles, easy cases and difficult cases. A pilot limited to the most enthusiastic volunteers risks overestimating future use.
Profiles matter. Brynjolfsson, Li and Raymond studied 5,172 customer support agents (Quarterly Journal of Economics, 2025). The assistant increases the number of issues resolved per hour by 15% on average. The gains in speed and quality are concentrated among the least experienced agents. So mix beginners and experts, then read the results by group. Finally, choose a duration that covers a full cycle of the activity, for example a monthly close. The explainer use case (in French) helps define the task being tested.
Organisation and assessments
Make organisational decisions with a clear assessment
Our assessments identify practical changes to help your teams and processes work together.
What should you measure during an AI pilot?
Measure four things: actual use, time spent, quality of the output and cost of verification. Read actual use in the tool’s logs, not in a survey. Have an expert judge quality, on a sample. The cost of verification counts the time spent reviewing and correcting: the explainer human in the loop (in French) describes this check.
Do not rely on how users feel. In July 2025, METR published a randomised trial of 16 experienced developers. With early-2025 AI tools, they took 19% longer. Yet they thought they had gained about 20%. METR now describes this result as outdated. Its February 2026 update suggests a speed-up, based on data METR considers very fragile. The methodological lesson holds: measure time, do not just ask for opinions.
How do you decide at the end of the pilot?
Compare the measurements with the criteria written at the start, then decide on the estimated value after all costs. The guide AI ROI explains how to cost these items. Three decisions are possible: extend, fix and retest, stop. An extension of the pilot without a new criterion is not a decision.
Fictional exercise. A mid-sized services company ran four ten-week pilots. The table shows the measured result and the decision taken.
Pilot | Measured result | Decision |
|---|---|---|
Meeting minutes, purchasing | 30% time saved, short review | Extend to the other sites |
Responses to complaints | Time saved, but one response in ten to rewrite | Fix the document base, retest for six weeks |
Stock-out forecasting | Error equal to the current method | Stop |
Legal research assistant | 3 active users out of 20 after one month | Understand the rejection before any extension |
What do you need to add to scale up?
Scaling up adds six items that the pilot hides: recurring budget, a designated owner, integration, security, support and training. Maintenance weighs heavily. Sculley and his co-authors (NIPS 2015) note that real-world machine learning systems commonly incur massive ongoing maintenance costs.
Training is also a legal matter. Article 4 of the AI Act has required AI literacy measures since 2 February 2025. Since 27 July 2026, Regulation (EU) 2026/1744 has relaxed this obligation. Deployers take measures to promote AI literacy among their staff, without having to guarantee a specific level. The explainers AI adoption (in French) and AI and skills (in French) detail how to support teams.
Should you consult the works council (CSE) before an AI pilot?
In companies with at least 50 employees, the employer consults the CSE on the introduction of new technologies (article L2312-8 of the French Labour Code). Judges assess case by case whether a pilot falls under this rule. On 14 February 2025, the Nanterre judicial court suspended AI tools presented as a pilot. It saw them as a first implementation, prohibited before the end of the CSE consultation. On 10 February 2026, the Paris judicial court annulled the CSE’s appointment of an expert. It ruled that a trial of Copilot 365, open to volunteers for four months, did not constitute a major project. We read this judgment on Judilibre (the French courts’ public case-law database); we know the Nanterre decision through a commentary by CMS Francis Lefebvre (secondary source).
Inform the CSE as soon as you design the pilot and describe its effects on work. The signal AI and the CSE (in French) revisits the Nanterre decision and two other suspensions from 2025.
Frequently asked questions
How long should an AI pilot last?
How long should an AI pilot last?
Long enough to cover a full cycle of the activity tested and get past the novelty effect. Set the end date and the stop criterion before you start. Extending requires a new criterion.
What is the difference between a POC and an AI pilot?
What is the difference between a POC and an AI pilot?
A POC checks that a solution works technically, often on prepared data. A pilot tests use in real work, with real users, and measures value. A successful POC tells you nothing about adoption.
How many users should an AI pilot include?
How many users should an AI pilot include?
Enough to represent the profiles of the intended rollout: beginners and experts, several teams, easy and difficult cases. The exact number depends on the task. A group made up only of enthusiastic volunteers risks overestimating future use.
What if the AI pilot is inconclusive?
What if the AI pilot is inconclusive?
First look for the cause: missing measurement, scope too narrow or use too low. Fix that cause and retest once, with a new date and a new stop criterion. If the second test is still inconclusive, stop.
Sources
RAND Corporation, J. Ryseff, B. F. De Bruhl, S. J. Newberry, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed, RR-A2680-1, 13 August 2024: 65 interviews, five causes of failure. Basis for the section on failures: RAND
E. Brynjolfsson, D. Li, L. Raymond, Generative AI at Work, Quarterly Journal of Economics, vol. 140, no. 2, 2025, pp. 889-942 (arXiv version of 6 November 2024, same figures): 5,172 agents, +15% issues resolved per hour, gains concentrated among the least experienced. Used for choosing the scope: arXiv
METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 10 July 2025: randomised trial, 16 developers, 19% more time, perceived gain of about 20%. Used for measurement: METR
METR, “We are Changing our Developer Productivity Experiment Design”, 24 February 2026: likely speed-up with late-2025 tools, based on data considered very fragile. The “outdated results” banner appears on the July 2025 page: METR
D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, NIPS 2015: ongoing maintenance costs of real-world systems. Used for scaling up: NeurIPS
French Labour Code (Code du travail), article L2312-8: consultation of the works council (CSE) on the introduction of new technologies: Code du travail numérique
Paris judicial court, judgment under the accelerated procedure on the merits, 10 February 2026, RG no. 25/57412: Copilot trial open to volunteer employees, “planned over a four-month period”. Basis for the sentence on the Paris decision: Judilibre
CMS Francis Lefebvre, C. Froger-Michon, Quand le CSE stoppe le déploiement de l’IA (When the CSE halts the AI rollout), 14 April 2025 (secondary source): Nanterre judicial court, 14 February 2025, no. 24/01457: CMS
CMS Francis Lefebvre, Copilot 365 n’est pas un projet important rendant nécessaire la consultation du CSE (Copilot 365 is not a major project requiring CSE consultation), 12 March 2026 (secondary source): Paris judicial court, accelerated procedure on the merits, 10 February 2026, no. 25/57412 (CMS wrongly writes “référé”, i.e. interim proceedings): CMS
CMS Francis Lefebvre, G. Loiseau, Intelligence artificielle : le forçage de la consultation du CSE (Artificial intelligence: forcing CSE consultation), 17 September 2025 (secondary source): overview of the 2025 decisions, read to check the context: CMS
Regulation (EU) 2026/1744 of 8 July 2026 and AI Act consolidated as of 27 July 2026, articles 4 and 113: AI literacy, applicable since 2 February 2025: EUR-Lex
The fictional exercise and the decision grid are a Spentia methodological construct, with no success statistics or client portfolio claimed.
