WER vs CER for Urdu ASR: How to Measure Speech Recognition Accuracy

The short answer

Use both. Word error rate (WER) is the number buyers and papers quote, so you need it for comparison. Character error rate (CER) is usually the fairer signal for Urdu, because Urdu spelling, spacing and Roman Urdu transliteration vary in ways that WER counts as full word errors. Neither metric tells you whether the meaning survived, so add a short meaning-critical check on top. And whatever you choose, apply the same text normalisation to the reference and the model output before scoring.

What WER measures

WER compares the model output with a verified reference transcript, word by word. It counts the minimum number of substitutions (S), deletions (D) and insertions (I) needed to turn one into the other, divided by the number of words in the reference (N):

WER = (S + D + I) / N

It is simple, widely understood and easy to compare across systems. Its weakness for Urdu is that a "word" is not stable. The same word can be written kya, kia or kiya in Roman Urdu. In Nastaliq, writers differ on spacing in compounds and on optional diacritics. Each of those differences counts as a whole wrong word, even when a native speaker would say the transcript is correct.

What CER measures

CER uses the same formula on characters instead of words. A one-letter spelling difference costs one character, not one word, so CER is far less sensitive to spelling variants. That is why it is common for languages with rich morphology or loose spelling conventions. Its weakness is the opposite of WER’s: a short but important word that is completely wrong, such as a number or a negation, barely moves the CER.

	WER	CER
Unit	Words	Characters
Effect of spelling variants	Large — each variant is a wrong word	Small — usually one or two characters
Effect of a wrong short word	Visible	Can be almost invisible
Comparable with published results	Yes, most papers report it	Often, especially for low-resource languages
Best use for Urdu	Headline number, after normalisation	Fairer accuracy signal, alongside WER

One sentence, two stories

Take a typical appointment request in Roman Urdu: “kya aap kal subah das baje aa sakte hain” (can you come tomorrow at ten in the morning?). Here are two model outputs. Both examples are illustrative, not client results.

Figure 1. Illustrative output A: four spelling variants. A native speaker reads it as correct.

Figure 2. Illustrative output B: one wrong word — aaj (today) instead of kal (tomorrow). The meaning is wrong.

Figure 3. Output A looks four times worse than output B on WER, yet A is usable and B sends the customer on the wrong day.

On raw WER, output A scores 44.4% and output B scores 11.1%. On CER, A scores 12.5% and B 5.0%. If you ranked models by raw WER alone, you would pick the one that books the wrong day. This is the core problem with scoring Urdu naively: the metric reacts strongly to spelling and weakly to meaning.

Normalise before you score

Most of the gap in output A disappears once both texts are normalised the same way. Agree the rules before testing and apply them to the reference and the hypothesis alike:

Script: decide whether you score Nastaliq, Roman Urdu or both, and never mix them inside one score.
Roman Urdu spelling: map common variants to one form (kia/kiya → kya, ap → aap). Keep the mapping list with the results so others can reproduce them.
Nastaliq: remove optional diacritics, unify character variants (for example different forms of yeh and kaf), and fix spacing rules for compounds.
Numbers: decide whether das, 10 and دس are equivalent, and score numbers separately if they matter to the product.
English words: in code-switched speech, write English in English script and score those spans separately. See our code-switching guide.

After normalisation, output A in the example drops to 0% WER, which matches what a native reviewer would say. Output B stays wrong — as it should.

Meaning-critical errors

Error rates treat every word the same. Products do not. For a bank, a delivery service or a clinic, a handful of word types carry most of the risk: numbers and amounts, dates and times, negations (nahin), names, and the intent words that route a call. We tag every error in these classes as meaning-critical and report their rate separately. Two models with the same WER can differ several-fold on this measure, and it is usually the number that decides whether a voice agent is ready.

Common Urdu ASR error patterns

When we read through failed Urdu utterances, the same patterns come up again and again. Tagging errors by pattern makes the report actionable for the team that has to fix the model:

Urdu-ization of English words: balance becomes bilans, reschedule splits into two words.
Accent substitutions: sounds that differ between Punjabi- or Saraiki-accented Urdu and broadcast Urdu are mapped to the wrong word.
Dropped short words: particles such as to, bhi and na disappear, sometimes reversing the tone of a request.
Number confusion: dhai (2.5), sawa (1.25) and paune (0.75) are misheard or written inconsistently.
Segmentation: compounds and verb phrases are joined or split differently from the reference.

Counting errors by pattern also shows whether a fix worked: after fine-tuning, the share of each pattern should fall, not just the overall WER.

Designing an Urdu test set

Who speaks: Urdu is a second language for most Pakistanis. Include speakers whose first language is Punjabi, Saraiki, Pashto or Sindhi, and record it, so results can be split by accent. Our Saraiki and Pakistani Punjabi tests are built this way.
What they say: mix read prompts with spontaneous answers, and include natural code-switching.
Where: phones and real rooms, not only quiet studios. Phone audio at 8 kHz is a different test from wideband microphones.
References: two native transcribers per clip, with disagreements resolved and logged. A noisy reference makes every metric lie.
Size: a first diagnostic of about 100 utterances shows where a model breaks; a few hundred per condition gives stable numbers.
Privacy: keep the test set held out and private, so it never leaks into training data.

What a good report shows

A single WER figure is not a report. Ask for:

WER and CER, before and after the agreed normalisation, with the normalisation rules attached.
Results split by accent or first language, recording condition and code-switch density.
A meaning-critical error rate, with examples.
Per-utterance results, so you can listen to the failures yourself.
A short list of the error patterns that matter most to fix.

This is the format of our Urdu speech evaluation and our voice agent and IVR audit. If you want the numbers on your own model, start with a fixed-scope pilot.

Questions

Is WER or CER better for Urdu?

Report both. WER is the comparable headline number; CER is fairer to Urdu spelling variation. Normalise the text the same way before computing either.

Why is my Urdu WER so high when the transcripts look fine?

Usually spelling and spacing variants, especially in Roman Urdu, are counted as wrong words. Apply an agreed normalisation and compare again; the remaining errors are the real ones.

What is a good WER for Urdu ASR?

It depends on the speakers, audio and domain, so there is no single target. Compare models on the same private test set and normalisation, and look at meaning-critical errors before deciding.

Should Roman Urdu and Nastaliq be scored together?

No. Score each script separately with its own normalisation rules, and never mix them in one number.

How many utterances do I need to evaluate an Urdu model?

About 100 well-chosen utterances are enough for a first diagnostic of where a model fails. Stable comparisons need a few hundred per accent or recording condition.