Revolab ASR Benchmark

Open evaluation of speech recognition models on real Malaysian audio — telephony, broadcast, short utterances, and public datasets

Best model per scenario

Scoring

WER = (Sub + Ins + Del) ÷ ref words lower is better · dual-reference: scores against both original and normalised transcript, keeps the lower per sample

Splits

Publicdisclosed test set Privateheld-out, checks overfitting Combinedweighted corpus WER across both

Thresholds

< 10% Excellent

10–20% Good

20–35% Fair

> 35% Poor

Loading results…

Loading…

Benchmark Methodology

How we evaluate Malay ASR models — dataset construction, scoring rules, normalization pipeline, and category definitions.

Dataset

Name

Revolab/ASR-Benchmark — hosted on Hugging Face

Language

Bahasa Malaysia (BCP-47: ms) — includes code-switching and dialectal variation

Splits

Publicdisclosed test set — used for public leaderboard Privateheld-out set — checks for overfitting to public

Audio format

16 kHz mono WAV; variable duration (0.5 s – 60 s); resampled to 16 kHz at eval time

References

Two references per sample: raw human transcript (text) and dataset-normalised form (normalized_text). Both are scored; the lower WER wins per sample.

Scoring

Primary metric

WER = (Sub + Ins + Del) ÷ N_ref Corpus WER: total errors over total reference words across all samples. Lower is better.

Error types

Sub — model output a different word (word confusion)

Ins — model hallucinated an extra word not in the reference

Del — model omitted a word present in the reference

CER

Character Error Rate — computed on normalised text; useful for morphologically rich Malay compounds. Reported alongside WER.

RTFx

RTFx = total_audio_s ÷ total_transcription_s Real-Time Factor (inverse). RTFx = 10× means 10 s of audio transcribed in 1 s. API models excluded — network latency dominates and is not representative of model speed.

Dual-reference

Each sample is scored against both the raw transcript and the normalised transcript. The lower WER reference is selected per sample before aggregating corpus WER. This accounts for legitimate transcription style variation.

Canonical WER

All WER scores on this benchmark are Canonical WER — computed after running both reference and hypothesis through the Malay canonical normaliser before alignment. This means legitimate word-boundary and spelling variants do not count as errors.

Examples of variants that collapse to the same canonical tokens:

takpayahlah · tak payah lah · takpayah lah→tak ya lah

okaylah · okeylah · ok lah→ok lah

tak ada · takde · takdak→tiada

sekejaplah · kejaplah→jap lah

bolehlah · boleh lah→boleh lah

sahaja · saja · aje→je

film · filem→filem

The normaliser handles: sentence-final particle fusion (lah, leh, kan, pun, mah, je, la, kot), possessive clitics (-ku, -mu, -nya), Indonesian vs Malaysian spelling, colloquial shortenings, fused compounds, and English loanword spellings. ~300 canonical mappings total.

Rare WER

Rare WER = (rare_Sub + rare_Del) ÷ rare_ref_words WER restricted to reference words not in the top 500 most frequent corpus words. Insertions excluded (no reference anchor). Isolates model weakness on tail vocabulary — proper nouns, technical terms, low-frequency Malay words. Shown in bar chart tooltips on the leaderboard.

WER thresholds

< 10% Excellent

10–20% Good

20–35% Fair

> 35% Poor

Text Normalisation

Pipeline

1. Strip noise/filler tags: <tag>, [tag], (tag)

2. Lowercase all text

3. Remove punctuation (keeps alphanumeric and whitespace)

4. Collapse multiple spaces

What is NOT done

Number expansion (e.g. "123" → "one two three"), abbreviation expansion, language-specific stemming. Numbers and abbreviations must be transcribed as the model produces them; both forms are accepted via the dual-reference mechanism.

Applied to

Both reference and hypothesis before WER/CER computation. Raw predictions are stored separately in the manifest.

Domain Categories

telephony

Live call-centre recordings; compressed 8 kHz audio upsampled to 16 kHz; background noise common

parliament

Malaysian parliamentary proceedings; formal register, long utterances, microphone setup

news

Broadcast news clips; standard newsreader Malay; clean audio with occasional music beds

drama

TV drama dialogue; scripted speech; emotion and overlap; studio + location audio

podcast

Conversational podcast audio; casual register, code-switching, multiple speakers

street interview

On-location vox pops; spontaneous, colloquial, heavy background noise

read-speech

Controlled read-aloud; clean studio recording; tests model ceiling on clean Malay

short-inputs

Single words or short phrases (< 3 s); tests model behaviour on brief commands and keywords

singing

Sung Malay lyrics; tests robustness to non-speech prosody and melody interference

animation

Cartoon/animated dubbing; stylised delivery, exaggerated prosody, non-adult voices

fleurs

Google FLEURS ms_my split; read sentences; open-domain vocabulary; public benchmark subset

commonvoice

Mozilla Common Voice ms clips; crowdsourced, diverse speakers and recording conditions

Leaderboard Groups

Telephony

Category: telephony. Represents deployment in call-centre and IVR scenarios.

Domains

All remaining real-world categories: parliament, news, drama, podcast, street interview, read-speech, singing, animation. Tests general-purpose Malay ASR quality.

Short Inputs

Category: short-inputs. Critical for keyword spotting, voice commands, and IVR DTMF replacement.

Open-Source

Categories: fleurs, commonvoice. Comparable to existing public benchmarks; useful for cross-benchmark calibration.

Reproducibility

Code

Evaluation harness available at github.com/Revolab-Sdn-Bhd/revolab-asr-benchmark ↗

Adding a model

Subclass BaseASRModel, implement transcribe_batch(), register in MODEL_REGISTRY. See CLAUDE.md in the repo for the full guide.

Batch size

Local models: batch size 4 (configurable via --batch-size). API models: single-sample sequential to respect rate limits.