Simpler, and still true: how we grade what Restate says

What is Restate?

Hundreds of millions of people have learned enough English to hold a simple conversation, yet much of what they would like to listen to is still above their level: a podcast, a lecture, a meeting at work, or the news. Restate, our product, listens along and says each sentence again in simpler English, a few seconds behind the speaker and in the speaker’s own voice. It works on whatever is playing. The listener picks the level and can change it while listening. You can hear it in the live demo, which restates the BBC World Service as it airs.

The Common European Framework of Reference for Languages, the CEFR, is the European scale for language learners. It has six levels, from A1 for beginners to C2 for near-native speakers. Restate offers three of them: A2, B1, and B2. A listener at A2 knows about two thousand everyday words and follows short, plain sentences. At B1, the vocabulary is wider, and a sentence can carry a second clause. At B2, speech can carry argument and nuance, with words that most learners meet only after years of study.

Below is one sentence of spoken English, followed by a restatement for each of the three levels. Each restatement is meant for a listener who knows less English than the one before.

The A2 line is clearly easier than the original. Whether it is a good restatement is a separate question, and it splits into three:

• Is it still true? The speaker said that our brains are “wired to prioritize” immediate rewards, a statement about what brains tend to do. In the restatement, our brains simply “want” good things, as if that were always so. The long-term goals have disappeared as well.
• Can a listener at A2 follow it? Every word, and both sentences, have to be within the level, because a listener cannot go back and re-read.
• Can it be said in time? One sentence has become two, and both have to fit into the time the original took.

Our metric, a program that reads each restatement and gives it a score, answers all three questions. It has two parts: code, for everything that can be checked exactly, and a large language model that judges the rest. This post explains how both work, reports what the code’s own checks show on 1,000 hours of restated speech, and comes back to this sentence for the metric’s verdict. If you work on text simplification or on the CEFR, we have four questions we would like your view on.

Why does a machine have to check this?

Restate produces a restatement like the ones in figure 1 for every sentence it hears. That is far more restatements than anyone could check by hand. The models behind it are trained on examples: thousands of hours of real speech in which every sentence is paired with simpler versions of itself. We generate those simpler versions with a large language model that works from written instructions. We call it the simplifier. A model trained on its restatements picks up whatever habits they have, including the bad ones. If some of the restatements silently drop a “not”, the trained model learns to drop negations.

To keep such habits out of the models, we use the metric at two points. Before training, we score each restated recording and leave out the ones that score badly. After training, we score what each new model produces before any listener hears it.

How do we grade a restatement when nobody has written the right answer? A teacher marking a test compares each answer with the correct one. For a restatement of yesterday’s lecture there is nothing to compare with, because no one has written the correct A2 version of it, and no one will. The metrics that researchers use for text simplification, such as SARI and LENS, assume that a correct version exists: they score a system by how close its output comes to simplifications that people wrote by hand. They also have no notion of a target level, so they cannot tell a sentence a beginner can follow from one that is merely a little simpler. Our metric asks each restatement the three questions above instead, and needs no correct version to compare with.

Can you spot what went wrong?

Before we describe the metric, here are four A2 restatements to judge for yourself. Three have a problem, each of a different kind, and one is fine. Each ruling names the error’s group (meaning, level, or delivery), its severity (critical, major, or minor), and its cost in penalty points. Make up your mind before you open a ruling.

The rulings follow the format that the metric uses. We borrowed it from MQM, Multidimensional Quality Metrics, a scheme in which reviewers of translations mark each error with a type and a severity. Every error falls into one of three groups, one for each of the three questions: meaning (is it still true?), level (can the listener follow it?), and delivery (can it be said in time, and the way a person would say it?). Every error also has a severity, defined by its effect on the listener, and each severity costs a set number of penalty points, so a restatement wants as few as possible:

• Critical, 25 points. The listener comes away believing something false, or cannot follow the sentence at all.
• Major, 5 points. The listener misses part of the point, or has to work to recover it.
• Minor, 1 point. The listener notices something odd and keeps following.

Because severity is defined by the harm to the listener, a major level error counts the same as a major meaning error. That lets us add the penalty points of all three groups into one score.

Most rulings of this kind come from the large language model, which we call the judge from here on. The rest come from the code, for everything that can be decided exactly. We check meaning first.

Is it still true?

The obvious way to check meaning is to ask a large language model whether the two sentences say the same thing. Models are good at that kind of judgment but unreliable on exact details: one can read straight past a changed number when the sentence around it is fluent. The code checks those details first. Before the judge sees a restatement, the code compares:

• every number in the original with the numbers in the restatement;
• every name, of a person, a place, or an organisation;
• every negation.

Anything that differs is passed to the judge as a fact, and the judge has to rule on it, because a difference is not always an error: “1,024” may become “about a thousand” without harm.

The rest of the meaning check needs judgment. The judge reads the original and the restatement, lists the meaning errors it finds, and gives each one a type:

• a distortion says something different from what the speaker said;
• an addition says something the speaker did not say;
• an omission leaves out something the speaker did say.

In SALSA, an earlier scheme for grading simplifications, people go through every change a simplification made to the original and mark the ones that hurt the meaning. We ask a language model to do that instead.

MQM was made for translation, and a translation has to keep everything that was said: every word left out or added is an error. A simplification cannot work that way, so the judge’s instructions allow two things that a translation reviewer would mark:

• Leaving out what matters least. A side remark, a word like “very”, a detail the point does not depend on. The simpler the level, the more may go. Leaving out a core statement is an error at every level.
• Naming what was implied (explicitation). When a long sentence is split in two, the second needs a subject. The second A2 sentence in figure 1 starts with “They”, a word the original does not have. That adds a word, not a fact. A made-up fact is still an addition.

The metrics that researchers use often misread exactly these two operations, as Barayan and colleagues found, which is one reason the judge has explicit rules for them.

Is it at the listener’s level?

A level limits three things:

• Vocabulary. Checked in code, against word lists. A list gives the same answer every time it is asked.
• Grammar. Checked by the judge, from a list of what the level allows.
• Sentence length. Checked in code, against a limit for the level.

Our word lists come from the publishers whose exams and dictionaries learners already use. Cambridge, Oxford, and Tokyo University of Foreign Studies each grade their lists by CEFR level, and we merged them. A word belongs to the lowest level at which any of them places it. Names and numbers do not count.

The code flags every word above the listener’s level. A flagged word is a level error unless the restatement itself explains it:

1. The judge quotes the words that explain it, from the same sentence or from the sentences just before it.
2. The code checks that the quoted words are within the level.

If there is no such explanation, the word is a level error. A word one level up is a minor error, and a word two or more levels up is a major error, because the further a word is above the listener’s level, the less likely the listener is to know it.

The rule leaves room for the hard word that a passage cannot do without. In a talk about the economy, replacing “economists” with “people” throws information away, so the simplifier keeps the word and explains it in easy words: “Most economists, people who study money, think that helped.” The list flags “economists” as two levels above A2. The judge quotes “people who study money” as the explanation, the code confirms that all four of those words are within A2, and there is no error.

For grammar, the judge checks against a list of what each level allows. At A2, for example, the list excludes the past perfect, reported speech, and “would” for unreal situations. The sentence-length limits are our own, set because a listener cannot re-read.

Can it be said in time?

A listener hears Restate’s output only once, at the speaker’s pace. Each restatement also has to fit into the time that the original sentence took. If a restatement runs long, it overlaps with the next sentence. If it runs short, the listener gets a stretch of silence. The code measures how long the restatement takes to say and compares it with the time the original took. A restatement that runs too long or too short for its slot is a delivery error, and the code reports it without any help from the judge.

Other delivery problems do need the judge. “It is fair to think this” is correct English, but it sounds stiff, because nobody talks that way. When a sentence is split in two, a pronoun in the second half can lose the thing it refers to. The judge reports problems like these as delivery errors. To catch them, it reads the three sentences before and the three sentences after the one it is judging, both as they were spoken and as they were restated, because the restated versions are what the listener hears.

Ormaechea and colleagues, who simplify transcripts of spontaneous French speech, stop at the written transcript. Our output has to be spoken, which is why delivery has rules of its own.

Who checks the judge?

A language model does not give the same answer twice. Ask one ten times whether leaving “long-term goals” out of the sentence in figure 1 changes the speaker’s point, and you will get both answers. We keep the judge in check in two ways.

The first is to tie it to the facts that the code has already established. Before the judge reads a restatement, it receives those facts: which words are above the level, and by how many levels, which numbers, names, and negations changed, and whether each sentence is within the length limit and the whole fits its slot. The words, numbers, names, and negations are open questions, and the judge must rule on each. The length facts are settled, and the code adds those errors itself. Figure 3 shows the facts for one restatement.

The second way is repetition, an idea from machine translation. GEMBA-MQM hands MQM marking to a large language model, and GEMBA-MQM v2 showed that ten independent judgments are more reliable than one. We use GEMBA-MQM v2 to grade our live speech translation and have written about it before. Our judge likewise reads every sentence ten times, each time from the same facts, and we call each reading a pass. The harshest pass is dropped as an outlier, and the rest are averaged with more weight on the lenient ones, because a harsh pass tends to flag doubtful cases as errors. Both ways make the judge more consistent. Whether its rulings match a teacher’s is a separate question, which we have not yet tested.

To keep the cost down, we judge a sample of each recording’s sentences. To score the recording, we add up the penalty points of the sampled sentences, divide by the number of words in their originals, and multiply by 100:

score = −(meaning + level + delivery), in penalty points per 100 words of the original

A perfect score is zero. A score of −1 means about one minor error per 100 words. People speak about 150 words a minute, so that is one minor error every 40 seconds. Because the three parts add up exactly to the total, a bad score always shows which of the three questions the restatement failed.

The word lists and the sentence counts need no model, so they are cheap enough to run over every sentence of a thousand hours of speech.

What happened on 1,000 hours of speech?

In September 2026, the simplifier restated 1,942 recordings, 990 hours of speech in all, at all three levels. Compared with the originals, the A2 versions use 21% fewer words, their sentences are 41% shorter and contain half as many subordinate clauses (clauses nested inside another clause), and 96 of every 100 of their words are within A2, up from 91 in the originals.

The simplifier never saw our word lists, so the lists measure its output independently. Every number in this section comes from the word lists, from idiom counts against the MAGPIE corpus, or from measures of sentence structure. None of them involves a language model, and anyone who runs the same tools on the same text gets the same numbers. We do not report the judge’s scores here, because we have not yet checked them against teachers’ judgments.

The A2 restatements use easier words

Figure 6 counts content words: nouns, verbs, adjectives, and adverbs. Words such as “the” and “of” are the same at every level, so they are left out.

In the A2 versions, the share of content words above B2 drops by more than half, from 5.2% to 2.3%, and idioms almost disappear (from 0.42 per thousand words to 0.10). The share of A1 and A2 words rises in every one of the 1,939 recordings long enough to measure, by a median of 9 percentage points, and most in the recordings whose originals had the fewest easy words. Counting every word as it occurs (a running-word count), except names and numbers, the share within A1 and A2 rises from 91.4% to 95.7%. That is higher than in figure 6 because the most common words, such as “the” and “of”, are almost all easy. Listening research suggests that a listener should know about 95% of the words to follow along. By our lists, a listener needs a B1 vocabulary to reach that share on the original speech, and an A2 vocabulary reaches it on the A2 restatements.

Sentences get shorter at every level

The average sentence has 12.7 words in the originals, 10.7 at B2, 9.8 at B1, and 7.5 at A2. The A2 versions have a third more sentences than the originals, and half as many subordinate clauses per main clause (0.26 compared with 0.52). This is the change a listener at A2 needs: short, plain sentences.

At B2, the vocabulary hardly changes

At B2, the vocabulary shares in figure 6 stay within one percentage point of the original. There is little to change: 97.5% of the running words in the original speech are already within B2. A B2 restatement mostly changes the sentences, splitting long ones and removing false starts, so they come out 16% shorter with 19% fewer subordinate clauses.

Word lists and sentence counts show how far the restatements moved toward a level, but not whether a restated sentence is still true. That is the judge’s work, and to show it we return to the sentence from figure 1.

How did the opening sentence do?

Is it still true? No, not quite. “Wired to prioritize” has become “want”, and what brains tend to do is now stated as a fact. “Long-term goals” has become “good things that come later”. The idea of waiting is still there, but the goals are gone. We count them as a minor distortion and a minor omission, because a listener still gets the speaker’s point.

Can a listener at A2 follow it? Yes. Every word in the A2 line is within A2, both sentences are short, and the grammar is simple.

Can it be said in time? Yes. The A2 line takes about as long to say as the original, spread over two sentences instead of one, so it fits the time the speaker took.

In this pass, the sentence gets two penalty points. For comparison, a restatement that had lost the “not” would have got 25.

Whatever you are listening to, Restate says each sentence again in simpler English. Before a new model reaches listeners, we grade its restatements in this way. We are opening Restate to teachers, creators, and schools first. If you teach English and want your students to keep up with the podcasts they care about, talk to us. If you are learning English yourself, the live demo is open now.

What we do not know yet

The metric has limits, and these are the ones we know about:

• Agreement with human raters. GEMBA-MQM v2 is trusted because its scores agree with those of professional reviewers of translations. We have not yet run such a study for simplification. The judge also varies: two runs over the same text give slightly different scores. We repeat a run before we believe a difference, and we rely more on the three part-scores and the list of errors than on the last digit of the total. - Bias toward similar models. The judge and the simplifier are large language models from the same family, and language-model judges are known to favour text that resembles their own. The word lists, the sentence-length limits, and the checks on numbers and names are the parts of the system that a model cannot influence. - Context across a recording. The judge sees only the neighbouring sentences. If a hard word is explained early in a recording and comes back twenty minutes later, the judge may count it as unexplained. - Word senses. The lists go by a word’s form, so “skeleton” the sport and “skeleton” the set of bones share one entry. Bannò, Knill, and Gales show that a language model with access to the English Vocabulary Profile can tell senses apart, one more reason to pair a list with a judge. - English only, for now. The method carries over to any language with graded word lists. The CEFR-J x28 project in Tokyo has extended its lists to 26 more languages, and Rabih, Qwaider, and Briscoe have worked out what it takes to write Arabic at a given CEFR level.

Four open questions

We would like your view on four questions:

1. Does the judge agree with teachers? A study with certified English teachers is our next step, and we would like to design it together with people who have run one before.
2. Should a major level error weigh as much as a major meaning error? We currently treat them as equal. Teachers could tell us whether listeners experience them that way.
3. What should we compare A2 speech against? Every set of level-graded sentences we know of is written text. On one such set, A2 sentences graded by teachers (Arase et al., 2022), 96% of the running words are within our A2 list, and our A2 restatements reach 95.7%. Both shares lean high, because our lists take the lowest level any publisher gives a word. Speech may need a target of its own.
4. How many unexplained words per hundred can a listener cope with? Listening research suggests about five, and we would like to measure this on restated speech.

If you are working on any of these questions, please write to dominik@voicefrom.ai.

Acknowledgements

This work was a team effort. In addition to the author, Abu Huzaifah Bidin, Mariam Elghandour, Muhammad Essam, Hassan Rom, and Halim Shukor contributed to the metric and the measurements described here.

None of this would have been possible without the people and organisations that back VoiceFrom. We thank our angel investors, our institutional investor Silicon Gardens, and Google, whose Google for Startups Cloud Program provided the generous cloud credits that paid for the computing behind this work.

References

Cited in the text

1. S. Ahrens, S. Deilen, S. Hernández Garrido, E. Lapshinova-Koltunski, and C. Maaß. 2025. Evaluation of machine translation errors in German plain language texts in the domain of health information. In Proc. KONVENS Workshops, 176-185. https://aclanthology.org/2025.konvens-2.13/

2. Y. Arase, S. Uchida, and T. Kajiwara. 2022. CEFR-based sentence difficulty annotation and assessment. In Proc. EMNLP. https://aclanthology.org/2022.emnlp-main.416/

3. S. Bannò, K. Knill, and M. Gales. 2025. Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs. In Proc. BEA. https://arxiv.org/abs/2506.02758

4. A. Barayan, J. Camacho-Collados, and F. Alva-Manchego. 2025. Analysing zero-shot readability-controlled sentence simplification. In Proc. COLING. https://arxiv.org/abs/2409.20246

5. P.-G. Barbu, A. Lipska-Dieck, and L. Lindner. 2025. EasyJon at TSAR 2025 shared task: Evaluation of automated text simplification with LLM-as-a-judge. In Proc. TSAR. https://aclanthology.org/2025.tsar-1.14/

6. A. Capel. 2010. A1-B2 vocabulary: Insights and issues arising from the English Profile Wordlists project. English Profile Journal 1, e3. https://doi.org/10.1017/S2041536210000048

7. T. Domhan and D. Zhu. 2025. Same evaluation, more tokens: On the effect of input length for machine translation evaluation using large language models. In Proc. EMNLP. https://arxiv.org/abs/2505.01761

8. M. Durbahn, M. Rodgers, M. Macis, and E. Peters. 2024. Lexical coverage in L1 and L2 viewing comprehension. Studies in Second Language Acquisition 46(4), 1045-1068.

9. D. Heineman et al. 2023. Dancing between success and failure: Edit-level simplification evaluation using SALSA. In Proc. EMNLP. https://aclanthology.org/2023.emnlp-main.211/

10. M. Junczys-Dowmunt. 2025. GEMBA V2: Ten judgments are better than one. In Proc. WMT. https://aclanthology.org/2025.wmt-1.67/

11. T. Kocmi and C. Federmann. 2023. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proc. WMT. https://aclanthology.org/2023.wmt-1.64/

12. A. Lommel, H. Uszkoreit, and A. Burchardt. 2014. Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica 12, 455-463.

13. M. Maddela et al. 2023. LENS: A learnable evaluation metric for text simplification. In Proc. ACL. https://aclanthology.org/2023.acl-long.905/

14. P. Nation and K. Wang Ming-tzu. 1999. Graded readers and vocabulary. Reading in a Foreign Language 12(2), 355-380. https://scholarspace.manoa.hawaii.edu/items/30b30b73-9201-4acb-a9b4-051c0a484b8b

15. A. O’Keeffe and G. Mark. 2017. The English Grammar Profile of learner competence: Methodology and key findings. International Journal of Corpus Linguistics 22(4), 457-489.

16. L. Ormaechea, N. Tsourakis, P. Bouillon, B. Lecouteux, and D. Schwab. 2025. Towards high-quality LLM-based data for French spontaneous speech simplification: An exo-refinement approach. In Proc. Interspeech, 4038-4042. https://www.isca-archive.org/interspeech_2025/ormaechea25_interspeech.pdf

17. A. Panickssery, S. R. Bowman, and S. Feng. 2024. LLM evaluators recognize and favor their own generations. In Proc. NeurIPS. https://arxiv.org/abs/2404.13076

18. N. Rabih, C. Qwaider, and T. Briscoe. 2026. Can LLMs control readability? A multi-dimensional evaluation framework for CEFR-controlled Arabic generation. In Proc. READI x TSAR at LREC. https://aclanthology.org/2026.readi-1.6/

19. Y. Tono. 2019. Coming full circle: From CEFR to CEFR-J and back. CEFR Journal 1. https://cefrjapan.net/images/PDF/CEFRJournal/CEFRJournal-1-1_D_Tono_2019.pdf

20. H. van Zeeland and N. Schmitt. 2013. Lexical coverage in L1 and L2 listening comprehension: The same or different from reading comprehension? Applied Linguistics 34(4), 457-479. https://doi.org/10.1093/applin/ams074

21. W. Xu et al. 2016. Optimizing statistical machine translation for text simplification. TACL 4, 401-415. https://aclanthology.org/Q16-1029/

22. Y. Yano, M. H. Long, and S. Ross. 1994. The effects of simplified and elaborated texts on foreign language reading comprehension. Language Learning 44(2), 189-219. https://doi.org/10.1111/j.1467-1770.1994.tb01100.x

Word lists, data, and tools we use

1. Cambridge University Press & Assessment. 2025. A2 Key and A2 Key for Schools vocabulary list. August 2025 revision. https://www.cambridgeenglish.org/images/506886-a2-key-2020-vocabulary-list.pdf

2. Cambridge University Press & Assessment. 2025. B1 Preliminary and B1 Preliminary for Schools vocabulary list. August 2025 revision. https://www.cambridgeenglish.org/Images/506887-b1-preliminary-vocabulary-list.pdf

3. Cambridge University Press & Assessment. 2025. Pre A1 Starters, A1 Movers and A2 Flyers wordlists. https://www.cambridgeenglish.org/Images/351850-a1-movers-word-list-2018.pdf

4. The CEFR-J Wordlist Version 1.6. Compiled by Yukio Tono, Tokyo University of Foreign Studies. Retrieved from https://www.cefr-j.org/download_eng.html in September 2026.

5. Council of Europe. 2020. Common European Framework of Reference for Languages: Learning, teaching, assessment. Companion volume. https://rm.coe.int/common-european-framework-of-reference-for-languages-learning-teaching/16809ea0d4

6. H. Haagsma, J. Bos, and M. Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proc. LREC, 279-287. https://aclanthology.org/2020.lrec-1.35/

7. X. Lu. 2010. Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics 15(4), 474-496.

8. Oxford University Press. The Oxford 3000 by CEFR level. Retrieved from https://www.oxfordlearnersdictionaries.com/external/pdf/wordlists/oxford-3000-5000/The_Oxford_3000_by_CEFR_level.pdf in September 2026.

9. Oxford University Press. The Oxford 5000 by CEFR level. Retrieved from https://www.oxfordlearnersdictionaries.com/external/pdf/wordlists/oxford-3000-5000/The_Oxford_5000_by_CEFR_level.pdf in September 2026.

10. Oxford University Press. The Oxford Phrase List. Retrieved from https://www.oxfordlearnersdictionaries.com/external/pdf/wordlists/oxford-phrase-list/Oxford%20Phrase%20List.pdf in September 2026.

Dominik Roblek

CEO & Founder

Dominik is CEO of VoiceFrom. He spent seventeen years at Google, most recently leading the Sense Audio team in Google Research, which shipped Now Playing on Pixel and noise cancellation in Google Meet. He co-authored Now Playing, Fréchet Audio Distance, SEANet, SPICE, and AudioLM.