How well do pronunciation APIs actually judge learners?

Why we did this

Say you are learning Japanese and read a sentence aloud. The app underlines one word in red. You practise that word and trust that the rest was fine.

That only works if the app is right. If it misses a real mistake, you keep making it. If it flags a word you said correctly, you start to doubt the feedback, and after a few of those moments many learners stop practising.

So how reliable are pronunciation APIs? Vendors make claims but rarely publish numbers you can compare, and almost never beyond English. We wanted an answer for every language we support, so we built an open benchmark that runs against any API, and we are publishing it.

The setup

We compared Lingolix, Microsoft Azure pronunciation assessment and TonePerfect, each in its documented best configuration, in October 2026. Two tests, because either one alone would mislead:

One note on fairness: we built Lingolix, so you should not take our word for any of this. None of the test recordings were used to train our model, and the test set and scripts are public, so you can rerun every number here yourself.

Results

The first chart shows how cleanly each system separates the planted errors from correctly spoken words. 1.0 is perfect; 0.5 is a coin flip.

Lingolix leads in all six languages. The gaps are widest in German, French and Spanish; in Japanese and Korean, Azure is close behind.

The number a learner actually feels is the false alarm rate: if a system is tuned to catch 80% of real errors, how many correctly spoken words does it flag on the way?

In five languages Lingolix keeps that to 1–3% of correct words. Japanese is the exception: there Azure is ahead, 3% to our 5%.

Real learners: a close race, and where Azure is ahead

On whole sentences Lingolix is close behind Azure; our sentence score is also built to give gibberish a 0, which costs a little agreement here. On single words Azure is clearly ahead: its word scores capture more of the “almost right” that human judges hear. That is the gap we are working on next.

Speed, measured from Berlin: Lingolix answers in about 0.8 seconds, Azure in about 1 second and TonePerfect in about 2.5 seconds per sentence.

Three things that surprised us

1. 1. Japanese length is make-or-break

   おばさん (aunt) and おばあさん (grandmother) differ only in how long one vowel lasts. It is one of the most common mistakes learners make. Of 40 planted length errors, Lingolix caught 37, Azure 36 and TonePerfect none. A system can be good at sounds and still miss the error learners make most.

2. 2. Native speech alone is not enough

   TonePerfect labels its packs for these languages experimental and scores against native reference speech; its Japanese and Korean packs say they are “not yet calibrated on graded learner recordings”. The results show what that costs: a learner's errors are not a native speaker's variation. To be fair, TonePerfect is built around Mandarin tones, which this benchmark does not test.

3. 3. Sentence scores are easier than word scores

   Every system agrees with human judges much better on whole sentences than on single words. Worth knowing before you show a learner a grade on every word: the sentence score is the more reliable number, and the word marks are best used to point at where to listen.

Limitations

• The errors in test 1 are planted, not made by real learners.
• Test 2 covers one language and one group of learners (native Mandarin speakers learning English).
• There is no public, human-rated learner data for German, Japanese or Korean. If you have some, let's add it.

FAQ

Get the data

The benchmark is meant to be rerun, including against us. The download contains the test lists and labels, the audio (or a script to fetch it where the license requires), the harness that calls each API and computes every number in this post, and a short guide with the commands, the cost of a full run and the setup traps for each API. Bring your own API keys.