Speaker Bias in African Language ASR: What AfriBED's Male-Speaker Gap Teaches Us About Balanced Test Sets

AfriBED found every fine-tuned ASR model it tested was biased against male speakers. Here's how to build a balanced bias test set for African languages.

A single minimal microphone split by soft purple and blue light, standing alone on a near-black background.

Most African-language ASR evaluations report one word error rate and stop there. AfriBED, a new bias evaluation dataset for Bemba and Nyanja, shows what that single number hides: in its baseline experiments, every fine-tuned model showed bias against male speakers. This post covers what the dataset found, why speaker bias in ASR for African languages rarely gets measured, and a concrete method for building a balanced test set for Twi, Wolof, Hausa or Yoruba.

What AfriBED is

AfriBED was submitted to arXiv on 19 October 2026 and accepted to the IEEE SLT 2026 demo track. It is an ASR bias evaluation dataset for two Zambian languages, Bemba and Nyanja. Its two subsets contain about 9 hours of speech from 80 speakers and about 18 hours from 127 speakers, for 207 speakers in total.

The speech is read, and the dataset is designed so that matched speech can be compared across gender and speaker-type groups (see the full text for how the authors define each group). That design choice is the important part. A bias dataset differs from an ordinary test set because it controls what is being compared. If one group reads harder sentences than another, any WER gap mixes speaker effects with content effects, and you cannot separate the two afterwards.

Fine-tuned models were biased against male speakers

The headline result goes against what most people expect. Discussions of ASR bias usually assume female speakers get worse recognition. In AfriBED's baselines, every fine-tuned model was biased against male speakers.

We will not guess at the cause. The paper is a dataset and demo contribution, so we don't treat it as a causal study. The practical lesson doesn't depend on the cause, though: you cannot predict the direction of a bias gap from priors. You have to measure it. A team that checked only for the gap it expected would have missed this one.

The result also shows why an aggregate WER says nothing about fairness. Suppose a model scores a respectable overall WER on a test set that is 70% female. Its weaker performance on male speakers is diluted almost out of sight. Change the deployment population to a call centre, a farmers' advisory line or a transport app where most users are men, and the "respectable" model becomes your worst-performing segment. The overall number didn't lie. It answered a different question from the one you needed answered.

Why speaker bias in ASR for African languages goes unmeasured

The research base is thin. A systematic literature review in ACM of bias evaluation and mitigation in ASR for low-resource African languages finds that this body of work is far smaller than the equivalent work on English. For most African languages, nobody has published a per-group error analysis at all.

Gender is also only one axis. Work on English has modelled gender and dialect bias together, on the reasoning that speaker attributes interact rather than adding up neatly. AfriBED splits by gender and speaker type, which is a solid start. But for languages like Twi, Yoruba or Fulfulde, dialect and regional variation may matter as much as gender, and these remain largely unmeasured. There are also good arguments for linguistically informed evaluation of multilingual ASR for African languages instead of treating WER as language-neutral. Tone, orthographic variation and agglutination all affect what an "error" means.

The stakes are practical. Language and accent bias in ASR has real consequences for the users who get misrecognised, and in African markets those users are often the majority of the people a product is meant to serve.

Why your standard test set can't measure bias

Most African-language test sets were built to measure accuracy, not fairness. Three structural problems get in the way:

  1. Unbalanced speaker groups. Crowdsourced and scraped corpora tend to over-represent whoever was easiest to recruit. A test set with 40 speakers from one group and 6 from another cannot support a confident claim about the smaller group.
  2. Content differs between groups. If group A read news text and group B read conversational prompts, any WER gap mixes speaker effects with text difficulty.
  3. Read-only speech. Read speech is cleaner, slower and more formal than what users actually say. A model can look unbiased on read prompts and diverge sharply on spontaneous or code-switched speech, where accent, speaking rate and vocabulary all shift.

AfriBED handles the first two. The third is a stated scope limit, which we return to below.

How to build a balanced bias test set

Here is the method we use, adapted for languages like Twi, Wolof, Hausa and Yoruba. None of it is exotic. Most teams simply skip it.

1. Define the groups before recruiting

Pick the attributes that matter for your deployment, such as gender, age band, dialect or region, and native versus second-language speaker. Define them in writing, including how each attribute is self-reported or verified. If dialect matters (it usually does for Yoruba and Twi), name the dialect clusters up front instead of reconstructing them from metadata later.

2. Set recruitment quotas per cell

Recruit to a grid, not to a total. If you cross gender with three dialect regions, you have six cells, and each needs enough distinct speakers to support a per-cell estimate. Our working rule is that a cell with only a handful of speakers is anecdotal, however many hours they recorded. Speakers are the unit of variance, not utterances. Ten hours from five people tells you about five people.

3. Use matched prompts across groups

For read speech, every group should read the same prompt pool, or statistically equivalent pools rotated across cells. This is the core of AfriBED's design and the cheapest control you can add.

4. Cover read, spontaneous and conversational speech

Matched prompts only work for read speech. For spontaneous and conversational speech, match the task instead: the same picture-description prompts, the same topics, the same role-play scenarios. Then report results per speech style. A model's bias profile on read speech should not be assumed to carry over to conversation.

5. Plan for code-switching

Wolof–French, Yoruba–English and Twi–English code-switching is how people actually talk. Tag code-switched utterances so you can report error rates on them separately. Decide in advance how the transcription convention handles switched spans, or your WER will partly measure annotator inconsistency.

6. Get native-speaker ground truth and measure its agreement

A bias test set is only as trustworthy as its transcripts. If one dialect's references were transcribed by non-native annotators, the apparent model gap may come from errors in the labels. Have every reference corrected by a native speaker of the relevant variety. Blind double-annotate a sample of each batch, compute inter-annotator agreement per group, and confirm that reference quality is uniform across cells before trusting any comparison.

How to report results

A single WER is not a bias report. At minimum, publish something like this:

MetricWhat it tells youNotes
WER/CER per groupAbsolute performance for each cellReport CER as well for agglutinative or tonal orthographies
Gap between groupsAbsolute and relative difference vs. best groupState which group is the reference
Confidence interval on the gapWhether the gap is distinguishable from noiseBootstrap by resampling speakers, not utterances
Speakers and hours per groupHow much evidence each cell rests onFlag cells below your minimum-speaker threshold
Breakdown by speech styleWhether bias changes from read to spontaneous speechInclude code-switched subsets separately

Resampling by speaker matters. Utterances from the same speaker are correlated, so resampling them independently produces confidence intervals that look narrower than the evidence supports. If you have the data, a mixed-effects model with speaker as a random effect is the more principled option, the same approach as modelling gender and dialect jointly rather than one attribute at a time.

What this means for training data

Evaluation finds bias. Training data is usually where it gets fixed. Three implications for your data mix:

  • Set speaker and dialect coverage targets in the training mix, not just hours. A thousand hours from a narrow speaker pool can generalise worse across groups than far fewer hours spread across many speakers and dialects.
  • Include natural speech. If production users speak spontaneously and code-switch, training only on read speech bakes in a style gap that will show up as a group gap wherever speaking styles differ.
  • Hold annotation quality constant across groups. Uneven transcript quality between dialects teaches the model the unevenness. Native-speaker correction plus blind double annotation with measured agreement is how you check for this, not something to assume.

This is the reason Afriklang captures read, spontaneous and conversational speech with speaker metadata. Every transcript is corrected by a native speaker, and a sample of every batch is blind double-annotated, with inter-annotator agreement at or above 95%. Our Wolof benchmark shows what targeted, native-validated data does for model quality. Per-group evaluation is how you check that the gain holds across groups and does not come from the majority group alone.

Honest limitations of AfriBED

AfriBED is useful, and also narrow, and its authors are explicit about its scope:

  • Two languages, one country. Bemba and Nyanja results do not automatically transfer to West African or Sahelian languages with different phonology, tone systems and contact languages.
  • Read speech only. This leaves out the conditions where we would expect the largest gaps: spontaneous speech, conversation and code-switching.
  • Sample size. With 207 speakers across two languages and several groups, per-cell counts limit how far the results generalise. The male-speaker finding is a strong signal that bias is present. It is not a universal law about direction.

None of this weakens the core lesson. It is a reason to run the same kind of audit on your own languages and speech styles.

Checklist: a speaker-bias audit before you ship

  • Group definitions written down (gender, age band, dialect, L1/L2) before data collection
  • Recruitment quotas per cell, with a minimum speaker count per cell
  • Matched prompts for read speech; matched tasks for spontaneous and conversational speech
  • Code-switched utterances tagged and transcribed under a fixed convention
  • Native-speaker corrected references for every variety
  • Inter-annotator agreement measured per group on a blind double-annotated sample
  • WER/CER reported per group, per speech style, with speaker-level bootstrap CIs
  • Gaps compared against your deployment population, not just the test population
  • Training-mix coverage targets updated based on the gaps you found

If you are building a per-group evaluation set for Luganda, Ewe, Fulfulde, Hausa, Wolof, Twi or Yoruba, the catalogue lists what we can deliver, including speaker metadata and natural-speech splits. If you would rather talk through cell design and quotas for your own deployment first, you can book a discovery call.

Sources

With your consent, we use analytics cookies (Google Analytics, PostHog) to see how the site is used and improve it. If you decline, we only count anonymous page views, without cookies. You can change your choice anytime under Cookie settings in the footer. Privacy Policy