Wolof sentiment analysis: our pipeline vs state-of-the-art LLMs
Wolof dominates Senegalese social media, yet it is nearly invisible to frontier models. We measured exactly how invisible and what validation by native speakers changes.
annotated Wolof comments in the source corpus
stratified gold-standard sample, dual native annotation
frontier LLMs evaluated zero-shot, temperature 0, identical prompts
tasks: 3-class sentiment and 7-class emotion detection
Headline results
- Afriklang90.0%
- Microsoft Phi-445.0%
- Llama 3.3 70B43.5%
- DeepSeek-R134.1%
- Claude Sonnet 4.632.5%
- Gemini 3.1 Flash-Lite31.4%
- GPT-4o Mini24.4%
- GPT-4o23.3%
- Claude Opus 4.819.4%
F1 macro, higher is better. LLMs evaluated zero-shot at temperature 0 on a 1,000-comment Wolof gold standard; Afriklang measured as native-speaker inter-annotator agreement on the same 1,000 examples.
Sentiment analysis full results
Three classes: positive, negative, neutral. The gap between the Afriklang pipeline and the best frontier LLM is +45.0 F1 points.
| System | N eval. | Hedging | Accuracy | F1 macro | 95% CI | Kappa | Polarity inversions | $/1K |
|---|---|---|---|---|---|---|---|---|
| Afriklanghuman-validated | 1,000 | — | 85.2% | 90.0% | [0.875–0.925] | 0.84 | — | $30 |
| Microsoft Phi-4 | 1,000 | 20% | 41.6% | 45.0% | [0.359–0.538] | 0.028 | 38.4% | ~$2.50 |
| Llama 3.3 70B | 1,000 | 15% | 41.6% | 43.5% | [0.347–0.523] | 0.006 | 43.7% | ~$3.50 |
| DeepSeek-R1 | 1,000 | 50% | 27.8% | 34.1% | [0.129–0.552] | 0.053 | — | ~$2.19 |
| Claude Sonnet 4.6 | 1,000 | 46% | 25.0% | 32.5% | [0.279–0.367] | -0.062 | 28.8% | ~$1.80 |
| Gemini 3.1 Flash-Lite | 1,000 | 52% | 23.3% | 31.4% | [0.270–0.362] | -0.021 | 25.2% | ~$0.08 |
| GPT-4o Mini | 1,000 | 60% | 17.4% | 24.4% | [0.201–0.286] | -0.063 | 22.8% | ~$0.60 |
| GPT-4o | 1,000 | 71% | 14.9% | 23.3% | [0.168–0.300] | 0.036 | — | ~$5.00 |
| Claude Opus 4.8 | 1,000 | 73% | 12.6% | 19.4% | [0.153–0.239] | -0.086 | — | ~$15.00 |
Hedging: share of predictions fleeing to "neutral", counted as errors. Kappa: agreement beyond chance negative values are worse than random guessing. Polarity inversions: positive comments read as negative or vice versa, the most costly error class for brand monitoring. $/1K: cost to annotate 1,000 comments at production API rates; open models were benchmarked on rate-limited free tiers that do not hold at production volume. "": not measured or not applicable.
Emotion detection 7 classes
Joy, sadness, anger, fear, disgust, surprise, neutral. Five of the six LLMs evaluated score 0% F1 on at least three of the seven emotions those capabilities simply do not exist in Wolof.
| Model | Joy | Sadness | Anger | Fear | Disgust | Surprise | Neutral | F1 macro |
|---|---|---|---|---|---|---|---|---|
| Afriklanghuman-validated | Ref. | Ref. | Ref. | Ref. | Ref. | Ref. | Ref. | 85.0% |
| GPT-4o | 0% | 0% | 50.0% | 0% | 0% | 0% | 40.0% | 12.9% |
| Claude Sonnet 4.6 | 16.0% | 9.6% | 4.6% | 0% | 4.9% | 5.4% | 41.8% | 11.8% |
| Claude Opus 4.8 | 8.9% | 4.4% | 2.9% | 0% | 0% | 0% | 59.6% | 10.8% |
| Llama 3.3 70B | 11.8% | 23.5% | 0% | 0% | 0% | 0% | 31.2% | 9.5% |
| GPT-4o Mini | 5.4% | 0% | 0% | 0% | 0% | 11.1% | 42.9% | 8.5% |
| Microsoft Phi-4 | 0% | 0% | 0% | 0% | 0% | 25.0% | 0% | 3.6% |
F1 = 0% on a class means the model never correctly identified that emotion. Afriklang defines the gold standard every model is scored against ("Ref."); inter-annotator agreement on the emotion task is 85.0% F1 macro, Cohen's kappa 0.81.
Why frontier LLMs fail on Wolof
These are structural failures, not prompt-engineering bugs. They reproduce systematically across every example carrying the same linguistic markers.
Wolof is missing from training data
Billions of English tokens, hundreds of millions of French and a few thousand Wolof tokens, mostly academic. Abbreviated, phonetic, code-switched social-media Wolof is rarer still.
French-Wolof code-switching derails models
Most comments mix Wolof and French in a single sentence. LLMs parse each part independently and miss the overall meaning.
Sarcasm reads as praise
"Sàmmleen seetu yi!" is an exclamation of indignation. Every LLM tested labels it positive; native annotators read the social register, not the surface markers.
Negation lives inside the verb
"xamul" (does not know), "bëggul" (does not want): Wolof negation is a verbal suffix, not a separate word. Models that miss -ul flip negative sentences to positive. Per-class scores show the asymmetry: every LLM but one reads negative Wolof markedly better than positive (Phi-4: 55.6% vs 34.3%) neither reliably.
Religious context inverts polarity
A comment invoking God's fire is severe moral condemnation. LLMs read religious vocabulary as positive markers the source of 15-20% of observed polarity inversions.
One comment, two readings
Wolof comment
naaaan j'déconne surtout fais pas ca xD
Native annotators
Negative
Claude Sonnet 4.6
Positive
Model reasoning
"j'déconne indicates the person is joking… xD confirms the laughing tone… overall positive sentiment"
Why the model is wrong
In this Wolof context, "fais pas ca" with an exaggerated "naaaan" is a real injunction masked by apparent humor. The native annotator reads the full social register, not surface linguistic markers.
"Free" LLM annotation is the expensive option
The metric that matters is not the cost per label it is the cost of reaching production-usable quality.
| System | F1 macro | Cost / 1,000 | Errors / 1,000 | Real cost at production quality | SLA |
|---|---|---|---|---|---|
| Afriklang pipeline | 90.0% | $30 | 174 | $30, guaranteed | Yes |
| Microsoft Phi-4 (best LLM) | 45.0% | ~$2.50 | 584 | $800+ in engineering time, no guarantee | No |
| GPT-4o | 23.3% | ~$1.20 | 851 | Does not converge on Wolof | No |
Open models were tested via a rate-limited free tier that does not hold at production volume; the $/1K column of the sentiment table shows production API rates, from ~$0.08 for Gemini Flash-Lite to ~$15.00 for Claude Opus 4.8. Prompt-engineering cycles on a language the model has never seen do not converge: an estimated 3-5 iterations plus ~10 hours of engineer time at $80/h exceeds $800 per 1,000 examples an industry-standard projection, not an empirically tested cycle, with no result guarantee and no SLA.
Built to be checked
Source corpus
700,000+ Wolof comments from YouTube, TikTok and Facebook pure Wolof and French-Wolof code-switched, annotated for 3-class sentiment and 7-class emotion.
Gold standard
1,000 comments, stratified by class and source with a fixed seed for full reproducibility. Two independent native Wolof annotators per example; disagreements resolved by a senior third.
LLM protocol
Zero-shot, identical Wolof-specific system prompt, temperature 0, strict JSON output: label, confidence and reasoning.
Statistics
F1 macro, so every class counts equally. 95% bootstrap confidence intervals over 1,000 resamples: Afriklang [0.875-0.925] and GPT-4o [0.168-0.300] do not overlap the gap is statistically significant.
Afriklang baseline
Inter-annotator agreement between two independent native annotators on the same 1,000 gold-standard examples used to evaluate every LLM: 90.0% F1 (kappa 0.84) on sentiment, 85.0% F1 (kappa 0.81) on emotion the upper bound any automated system can reach on this data.
Known limitations
We publish these because a benchmark you cannot interrogate is marketing, not measurement.
- Inter-annotator agreement and model F1 are not identical measurement types: the Afriklang 90.0% measures the internal consistency of the human annotation process, while LLMs are scored against the adjudicated labels. IAA is the upper bound any automated system can reach on this data a model cannot systematically exceed the agreement of the humans who define the ground truth.
- The $800+ real cost of iterating an LLM toward production quality is an industry-standard projection (3-5 prompt cycles, ~10 engineer-hours at $80/h), not an empirically tested iteration cycle.
- Zero-shot only: few-shot prompting would lift LLM scores but it requires exactly the annotated Wolof data this pipeline produces.
Get the full benchmark report and a sample of the corpus
Or email us at contact@afriklang.com