Every dataset and evaluation follows the same documented pipeline, with written guidelines, two-reviewer verification and measured error rates in every delivery.
Local coordinators recruit by district, dialect and the mix agreed with you.
Purpose and uses explained in the speaker's language; signed and indexed.
Guided sessions with level, clipping and silence checks.
Native transcription plus independent second review.
Spelling convention, normalisation, tags and edge cases, versioned per project.
A senior linguist resolves conflicts; each decision is logged with a reason.
Guideline changes are dated and applied to all affected files.
| Metric | Definition | Why it matters |
|---|---|---|
| WER | Word substitutions + deletions + insertions ÷ reference words | Standard, comparable recognition accuracy |
| CER | Same at character level | Fairer for languages with variable spelling |
| Meaning-critical error rate | Share of utterances where the intent changes | Captures failures WER hides |
| Subgroup breakdown | Metrics split by dialect, condition, age band, code-switch | Shows who your model fails |
| Exclusions | Clips removed and why (e.g. unusable audio) | Keeps results honest and reproducible |
Errors found by you or our QA are corrected and redelivered.
Clips failing audio or consent checks are re-recorded with new speakers.
We agree acceptance criteria up front and measure against them.
A full specimen report will be downloadable at launch. The structure is below.
Illustrative example of our format. Live consented samples publish at launch.
A fixed-scope pilot across Saraiki, Pakistani Punjabi and Urdu with a full error report. You keep the findings.