Published
Updated
This guide sets out the criteria for choosing a language model for business use, and a way to compare them. It does not rank any provider or model. It assumes you have chosen the use: the guide to prioritising AI use cases covers that choice.
Key points
Choose a model for a use, tested on your own examples, and not from a public leaderboard.
Also compare cost, jurisdiction, reversibility and the model’s lifecycle, not only the quality of the answers.
Read the licence of an open-weight model: “open” does not mean “unconditional”.
Where do you start when choosing an LLM?
Start by describing the use and the data, before looking at models. An LLM (in French) suited to summarising confidential contracts is not necessarily suited to a multilingual customer assistant. For each use, note the task, the language, the expected volume, the sensitivity of the data and the acceptable response time. These five elements rule out options before any test.
Why test your own examples rather than a leaderboard?
A public leaderboard (in French) measures an average over tasks that are not yours, and its own biases sometimes distort it. In 2025, Singh and co-authors, mostly researchers at the vendor Cohere, studied the Chatbot Arena leaderboard, which became Arena in January 2026. They argue that private testing and selective publication of scores distort it. Arena disputes several of their figures. According to the authors, models end up adapting to this leaderboard rather than to general quality.
Instead, build a set of a few dozen real examples, in French if your users write in French. For each one, write the expected answer or the criteria for a good answer. Have business experts score the results without telling them which model answered. Keep this set: you will use it at every change of model.
Quality in French is judged on your own texts. Check your industry’s vocabulary, legal or technical phrasing and the tone your customers expect. Also test real documents: tables, emails, poorly written minutes. A model that shines on clean texts can disappoint on your archives.
How do you estimate the cost of using an LLM?
The cost of use depends on the number of tokens (in French) processed, as input and output, multiplied by the volume of requests. Measure it on your set of examples rather than estimating it. Language matters. According to Petrov and co-authors (NeurIPS 2023), the same text can contain up to fifteen times more tokens in some languages than in English. For French, the gap was about 1.6 times with the GPT-4 tokenizer. Add the cost of human verification and integration. The guide to AI ROI explains how to calculate the estimated value after all costs.
Organisation and assessments
Make organisational decisions with a clear assessment
Our assessments identify practical changes to help your teams and processes work together.
What should you ask about confidentiality and jurisdiction?
Ask the provider where it processes your data, whether it retains it, whether it uses it to train its models and which law applies to it. Server location is not enough. The US Cloud Act (in French) covers providers of electronic communication or remote computing services that fall under US jurisdiction. When a US authority issues a demand, it requires them to preserve or disclose the data they hold or control, wherever it is located (18 U.S.C. § 2713).
For the most sensitive data, the SecNumCloud (in French) qualification from ANSSI (the French national cybersecurity agency) aims in particular to withstand an order issued under extraterritorial laws. It is valid for three years, with annual surveillance audits. ANSSI publishes the list of qualified offerings: check that the specific offering you are targeting appears on it.
How do you limit dependence on a provider?
Limit vendor dependence (in French) by keeping your set of examples, your prompts and your data outside the tool, and by having a second model tested. Models change quickly. According to their documentation consulted in October 2026, OpenAI announces at least six months’ notice for its generally available models. Its specialised variants get at least three months. Its previews can disappear with much shorter notice, for example two weeks. For all its models, the vendor can shorten these periods for safety or compliance reasons. Anthropic announces at least 60 days’ notice for its public models.
These periods do not always cover the time needed to retest and redeploy a use. Write the notice period and access to dated versions into the contract.
Should you choose an open-weight model?
You can host an open-weight model (in French) wherever you wish, within the limits of its licence. You then take charge of its operation and security. Read its licence before any test. Meta’s Llama 4 licence requires you to request a licence above 700 million monthly active users at the release date. Its acceptable use policy does not grant rights to the multimodal models to individuals domiciled in the European Union, or to companies whose principal place of business is there. Yet Scout and Maverick, the published Llama 4 models, are natively multimodal. This restriction does not apply to end users of a product that integrates them. Mistral AI publishes most of its open models under the Apache 2.0 licence, which allows commercial use. Some, under a modified MIT licence, require a commercial licence above 20 million dollars in monthly revenue. Others fall under a research or non-production licence.
Which grid should you use to compare LLMs?
Compare the models on the same grid, criterion by criterion, then decide according to the weight of each criterion for your use. This grid is a Spentia proposal, with no success statistics or client portfolio claimed.
Fictional exercise. A mid-sized company wants to summarise supplier contracts in French. It compares three anonymous options on its set of 60 examples.
Criterion | Option A: API from a US vendor | Option B: API from a European vendor | Option C: open model hosted in-house |
|---|---|---|---|
Quality scored by the lawyers | 8.1 out of 10 | 7.8 out of 10 | 6.9 out of 10 |
Provider’s jurisdiction | US law | European law | In-house |
Withdrawal notice in the contract | Not negotiated | Negotiated | Not applicable |
Operating effort | Low | Low | High |
Decision | Rejected: confidential contracts | Selected for a pilot | Kept as a fallback |
Frequently asked questions
Mistral or ChatGPT: which should a business choose?
Mistral or ChatGPT: which should a business choose?
Neither wins in every case. Test them on your own examples, then compare the measured cost, the provider’s jurisdiction, the terms of use for your data and the withdrawal notice. The right choice depends on the weight you give each criterion for the intended use.
What is a sovereign LLM?
What is a sovereign LLM?
The term has no single official definition. We take it to mean a model operated by a provider subject only to European law, with data hosted in the European Union. Check three points: the law that applies to the provider, where the data is processed and, for sensitive data, a SecNumCloud-qualified offering.
Can you host an LLM on your own servers?
Can you host an LLM on your own servers?
Yes, with an open-weight model whose licence allows it. You gain control over the data and the version. You take charge of the servers, security, updates and monitoring.
Should you change models with every new version?
Should you change models with every new version?
No. Run your set of examples on the new version and switch only if the result or the cost improves clearly. A withdrawal announced by the provider, however, forces you to migrate before the shutdown date.
Sources
S. Singh et al. (Cohere, Cohere Labs and universities), The Leaderboard Illusion, arXiv, version of 12 May 2025: biases of the Chatbot Arena leaderboard according to the authors. Used for the limits of public leaderboards: arXiv
Arena, response to The Leaderboard Illusion, 9 May 2025: effect of private testing described as “minimal”, several figures disputed: arena.ai
A. Petrov, E. La Malfa, P. Torr, A. Bibi, Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023: tokenisation gaps by language, up to a factor of 15 (Shan); about 1.6 for French with the GPT-4 tokenizer (table 1): arXiv
18 U.S.C. § 2713 (Cloud Act): on demand, preservation or disclosure of data the provider holds or controls, regardless of where it is stored: Cornell LII
ANSSI, FAQ on the SecNumCloud qualification: aim of resisting extraterritorial laws, three-year validity, annual audits, list of qualified offerings: ANSSI
OpenAI, Deprecations documentation, consulted on 26 October 2026: minimum notice periods by model type, and exceptions: OpenAI
Anthropic, Model deprecations documentation, consulted on 26 October 2026: notice of at least 60 days for public models: Anthropic
Meta, Llama 4 Community License, Llama 4 Maverick model card (5 April 2025): threshold of 700 million monthly active users: Hugging Face
Meta, Llama 4 Acceptable Use Policy: multimodal models excluded for individuals domiciled in the EU and companies whose principal place of business is there, except end users; Scout and Maverick are natively multimodal: Meta
Mistral AI, help centre, open model licences, dated 12 August 2026: Apache 2.0 and modified MIT licence with a revenue threshold: Mistral AI
Mistral AI, Models documentation, consulted on 26 October 2026: open model licences (Apache 2.0, modified MIT licence, CC BY-NC 4.0): Mistral AI
The grid and the fictional exercise are a Spentia methodological construct, with no success statistics or client portfolio claimed.
