From idea to pilot: running an AI project you can measure

Many organisations have already “tried AI”. An assistant connected to a few documents, a convincing demo in a meeting, then nothing. The pilot did not fail, strictly speaking. It simply never led to a decision.

That is the symptom I come across most often: an AI project started from a technology rather than from a question. People ask what a language model could do, instead of asking which problem deserves solving, and how they will know it is solved.

Here is the approach I follow, shaped by concrete projects: a benchmark for reading supplementary health insurance cards, a conversational coach built into a training app, and agents that prepare part of my own operations.

Start from the decision, not the model

A useful pilot ends with a decision: extend, adjust or stop. If nobody knows at the outset which criteria that decision will rest on, the pilot will produce an impression, not an answer.

So before choosing a model, I write down three things with the client. The precise task the system must perform, in business terms: read this field on this document, answer this category of question. The success criterion, measurable on real cases: a share of correct fields, a processing time, a share of acceptable answers. And the stop condition: what, if observed, means the approach does not fit.

The last one is the hardest to write, and the most useful. A pilot that cannot fail cannot demonstrate anything.

Build the means to compare before you choose

The second trap is choosing a tool before having a way to judge it. For a task like document reading, there are dozens of options: hosted OCR services, general-purpose vision models, open-weight models you can run yourself. Each has its strengths, costs and confidentiality constraints.

For Santécie, rather than plugging in the best-known service and judging it on a few examples, I first built a benchmark. Every pipeline is versioned and produces artefacts that never change. Two engines, or two settings of the same engine, can be compared on the same images without wondering whether something shifted between runs.

This work looks slow at first. In practice it speeds up everything else: each new option can be tested in hours, and the discussion is about results, not sales promises.

Treat confidentiality as a design constraint

With sensitive data, “where is the data processed?” is not a detail to settle after the pilot. It determines which options exist at all.

For the health insurance cards, processing is designed to stay on the user’s machine. An external API is only considered for benchmarking, and only with a zero-retention guarantee. Part of the work was analysing OCR services hosted under French health-data hosting (HDS) rules, their confidentiality commitments, and the conditions under which an open-weight model could be served on controlled infrastructure.

That framing rules some solutions out from the start. That is exactly the point: the pilot will not end on an option that compliance rejects afterwards.

Bound what the model can do

A language model plugged in without limits quickly produces answers that sound right and are wrong. On subjects close to health or money, that is not acceptable.

In the training app built for Essan NFC, the conversational coach does not answer “from memory”. It retrieves the user’s context through read-only commands, works in bounded search rounds, and cannot change anything. It relies on what is known about the user and the method, not on generalities. Its advice also stays informational: it does not replace medical advice, and the app says so.

The project also explored running a compact model directly on the phone, for privacy and offline use. That track stayed a development experiment: it measured the real constraints (size, speed, quality) without forcing them on the product. That is exactly what exploration is for: learning something that could change a decision, without turning into an implicit commitment.

Put human validation inside the process

An AI pilot that works immediately raises another question: who is responsible for what the system produces?

My answer does not change: the system prepares, a person validates. On the card reading demonstrator, every proposed field is reviewed by a person before confirmation. In my own operations, agents draft invoices and emails, but nothing goes out without my approval. That validation is not a button added at the end: it is a step of the process, designed from the start and checked against specific data.

This choice has a welcome side effect on measurement. Every human correction is a data point: it shows where the system gets things wrong, and how often. The pilot thus produces, as it runs, the indicators needed to decide whether to extend it.

Write the limits as clearly as the results

The last point is about honesty, and credibility. A pilot produces partial results. Presenting them as final sets up disappointment.

So every pilot summary I write has a section on limits: what was not tested, the cases where the system fails, the assumptions still to verify. On the OCR benchmark, for instance, the documentation states plainly that it is a demonstrator, with no claim to production quality or medical compliance. That transparency does not weaken the project. It lets the client decide with full knowledge.

The approach in short

1. State the decision the pilot must enable, with a success criterion and a stop condition.
2. Build the means to compare before choosing a model or a vendor.
3. Set the confidentiality frame from the start, so you do not explore unusable options.
4. Bound the model: controlled context, read-only access, explicit scope.
5. Build human validation in as a step of the process, and use it as a source of measurement.
6. Document the limits as carefully as the results.

A measurable AI project is not slower than any other. It simply avoids running the same inconclusive pilot again six months later.