You can't judge an AI feature by how often it's used, but by what users think of what it produces. Edifice checked this with Screeb: the satisfaction rate of its rewriting assistant, based on the FALC technique (Facile à Lire et à Comprendre, the French Easy Read standard), rose from 34% in June to 79% in early October, by adjusting the prompt and the LLM based on feedback.
Edifice builds educational platforms used by primary and secondary schools across France. Romain Gentet, Lead Product Manager at Edifice, set up the measurement system. He explains how a simple survey, shown a few seconds after each generated text, let the team know what worked, what didn't, and in which direction to correct the AI.
Why measuring AI satisfaction changes everything
With a classic feature, you know what it will produce: the result depends on an algorithm you control. With generative AI, the result varies from one use to the next, and its quality is partly subjective: a text can be correct on paper and still not suit the person who has to publish it.
For Romain, the issue isn't that measurement is harder, it's that it becomes more important:
“ I wouldn't say an AI feature is harder to measure. Of course there are always classic service-quality metrics, but measuring satisfaction matters more because we don't control exactly what the AI will produce, unlike a more algorithmic, classic feature. The iterative approach seemed more relevant from the start, and also a good way to move faster to production. ”
— Romain Gentet, Lead Product Manager, Edifice
The Edifice case: a FALC AI to speak to every family
Edifice builds educational platforms that regions and departments use to communicate with families. In Hauts-de-France, the platform covers the region's high schools and schools, an audience of around 400,000 parents.
For these institutional messages, Edifice added to its text editor an AI rewriting feature inspired by FALC, “facile à lire et à comprendre” (easy to read and understand), an inclusion approach of European origin. The author writes their text, and the AI suggests a simpler version. It has been in production since March 24 in Hauts-de-France, and has since been rolled out to the other platforms.
The challenge comes down to two things:
- Acceptability: the suggested text must convince the person who will sign and send it.
- The prompt: the text must be understandable, without overly choppy sentences, in the service of inclusion.
For this kind of message, there's no way to know whether the AI holds up without asking the people who use it.
The setup: a Screeb survey right after the generated text
Edifice configured in Screeb a short in-app survey that appears in the application about 10 seconds after the AI returns its text. It asks a first question, “ What do you think of the generated text? ”, with a thumbs up or a thumbs down. Two open questions follow: “ What could be improved? ” and a free comment.
The right moment: just after the text has been read
The trigger is the decision that matters most. Romain explains:
“ It seemed more relevant to do it right after the suggested text is read, to get a better response rate and feedback as fresh as possible, so that the feedback on the wording is as relevant as possible. Also, users need to clearly see what the survey is about, because we have dozens of other features in the platform. An asynchronous form was less suited to this case. ”
— Romain Gentet, Lead Product Manager, Edifice
A questionnaire sent later would have asked the user to remember one text among dozens of uses. Here, the question arrives when the opinion is freshest.
A format that limits friction
According to Romain, choosing a thumbs up or down followed by two open questions was a way to generate the most engagement in the survey flow with a simple question. It's also a pattern that is starting to take hold in AI rewriting features. Users are getting used to it, which reduces friction even further.
The before and after text, attached to each response
The survey doesn't only send back a score. Edifice collects, in a pseudonymized way, the text entered by the user and the version suggested by the AI, attached to their response. This feedback is then exported to an agent that runs a semantic analysis, to check that the rewriting holds up against what users say.
When a thumbs down comes in, the team can read what the user had written, what the AI returned, and what they think of it.
What the feedback taught the team
From the first weeks, the feedback highlighted a recurring area for improvement: the tone, judged too familiar. Comments were short and concrete, such as “ too familiar”, “ the level of language” or “ the formal 'vous'”. The same finding came up in parallel through a client department, for whom using the formal 'vous' is a recommendation from the French Ministry of National Education.
A level of formality that depends on who writes
By reading the before and after texts, the team understood that the expected register wasn't the same for everyone:
“ We realized that the expected level of formality could depend on the type of communication and the user's profile. A school principal will expect a more formal text. A teacher, when communicating generally, usually adopts a friendlier tone. ”
— Romain Gentet, Lead Product Manager, Edifice
First surprise: teachers used the feature more than expected, even more than school principals, for their internal communications. The inappropriate tone was therefore not only a matter of improving the prompt, but also of the user's profile and audience.
Iterating on the prompt and the model
For Edifice, the rise in satisfaction comes from a series of adjustments to the prompt and to the LLM used behind the feature. The feedback helped find the balance between the FALC format and how users actually write, with a good reason to proceed by trial: each version could be tested, then validated or dropped based on data rather than intuition.
This is where Screeb changes the way of working: the team makes decisions with real data, surfaces metadata useful for analysis, and saves time to test and iterate.
Results: from 34% to 79% satisfaction
The satisfaction measured by the survey went from 34% in June to 79% over the first half of October.
It's a simple indicator. It measures users' approval of the suggested text, not how well families understand it. Also, only users who respond are counted: at launch, about one in three gave their opinion, with a possible over-representation of the least satisfied. However, the measurement method stays the same from one version to the next, which makes it possible to compare AI versions and identify the one that collects the most favorable feedback.
Public sector: hosting in France, a mandatory requirement
When a measurement tool collects texts written by local authority staff or teachers, the question of data hosting arises from day one. For Edifice, the answer fits in one sentence:
“ Hosting in France was simply a mandatory requirement for us. ”
— Romain Gentet, Lead Product Manager, Edifice
This requirement didn't stay theoretical: Edifice's CISO (RSSI) reviewed Screeb's legal terms before the team approved the subscription. On Screeb's side, the guarantees come down to a few points:
- Hosting: 100% in France, with Scaleway and OVH.
- Jurisdiction: outside the CLOUD Act.
- Encryption: all exchanges are encrypted.
- Compliance: GDPR, ISO 27001 and the AI Act.
- AI: feedback analysis relies on a French AI, hosted in Europe (Mistral AI).
At Edifice, the texts collected to understand the feedback are pseudonymized.
For teams working with the public sector, getting the hosting and the legal terms of the measurement tool validated upfront is essential, before configuring the first questionnaire.
Measuring the quality of your AI with Screeb: the Edifice method
Since this first experience, the method has been reused several times at Edifice. Romain's advice to a team launching an AI feature comes down to one idea:
“ Thinking about complex data to measure the adoption rate or satisfaction through navigation or usage data isn't always the most relevant. In some cases, asking users directly for their opinion is more useful and faster to set up. ”
— Romain Gentet, Lead Product Manager, Edifice
Each step relies on a Screeb feature:
- Trigger the survey on a product event. Send Screeb an event at the right moment (for example “AI result displayed”), and the survey appears right after, with the targeting and frequency rules that avoid over-soliciting users. It can also be launched directly from your code, at the exact moment you want.
- Keep a light format with the survey builder. One rating question, then one or two open questions: few steps to configure, little friction for the user.
- Link the response to what the AI produced with hidden fields. Pass the input text and the generated text when the survey launches: they attach to each response, without the user seeing them.
- Read verbatims by theme with content analysis. Tone, language level, length: comments point to what needs adjusting. Screeb's Content Analysis surfaces the recurring themes in your feedback, with example responses for each.
- Change, then measure again in the survey results. Change the prompt or the model, and track satisfaction month after month on the same survey. If you send the prompt or model version as a property, you can also segment the results to compare before and after.
FAQ: measuring the satisfaction of an AI feature
How do you measure the satisfaction of an AI feature?
By asking users directly in the application, right after the AI result is displayed. Edifice uses a Screeb survey with a thumbs up or thumbs down, followed by open questions to understand what can be improved.
What is the right moment to trigger a survey on an AI feature?
A few seconds after the AI's response. At Edifice, the survey appears about 10 seconds after the generated text, which gives fresh feedback that is unambiguously tied to the right feature.
Do you need complex usage metrics to judge the quality of an AI?
Not always. According to Romain, asking users directly for their opinion is in some cases more useful and faster to set up than building adoption measures from navigation data.
How do you link a satisfaction score to what the AI produced?
By attaching to each response the text entered by the user and the generated version. Edifice collects them in a pseudonymized way, which makes it possible to understand why a result is liked or disliked and to improve the prompt.
Can you measure the quality of an AI in the public sector?
Yes, as long as you meet the data requirements. For Edifice, hosting in France was a mandatory requirement, and its CISO reviewed Screeb's legal terms before approving the subscription. Screeb hosts its data 100% in France, with Scaleway and OVH.
Conclusion: measuring is already improving
In a few months, Edifice went from 34% to 79% satisfaction on its FALC rewriting AI, relying on a simple measurement: a Screeb survey shown at the right time, the before and after texts to understand each piece of feedback, and prompt and model adjustments guided by users. All within a public-sector framework, with hosting in France required from the start.
Are you building an AI feature too? Measure its quality from day one with Screeb, and let your users tell you how to improve it.
.png)
%20(1).png)