INCLUDE Benchmark Records Bengali Highest Bias in Open-Source LLMs Across 14,988 Scores
The INCLUDE benchmark quantifies cross-lingual safety failures in LLMs through 2,604 prompts and 14,988 bias scores. Bengali shows peak bias in open-source models while English bias reverses in closed-source models. This exposes English-centric alignment limits for Indian language deployments.
The paper releases INCLUDE, a 2,604-prompt multilingual benchmark covering English, Hindi, Bengali, Marathi, Tamil, and Hinglish. Ten open- and closed-source LLMs were scored on Indian socio-cultural bias, generating 14,988 measurements. The design targets spoken dialogue systems serving India's linguistically diverse population where English-centric safety training leaves non-English outputs unfiltered.
Statistical results isolate two patterns. Bengali produced the peak bias mean among open-source models. English recorded the lowest bias in open-source systems yet the highest bias in closed-source systems. These reversals indicate that safety alignment strength does not transfer uniformly across languages or model access tiers.
The gap directly affects voice assistants and code-mixed dialogue systems now entering Indian markets. Existing English-only refusal training leaves stereotype propagation routes open in Bengali and Hinglish. Deployment records from multilingual regions show user-facing outputs bypassing filters that pass English test suites.
Future work requires language-specific safety data and evaluation loops that measure refusal rates per language rather than aggregate English proxies. Regulators and deployers must adopt benchmarks like INCLUDE before scaling spoken systems to low-resource languages.
AXIOM: Open-source models will require Bengali-specific refusal datasets exceeding 10k examples before bias scores drop below English baselines by Q4 2027.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2608.18131)
- [2]Supporting Source(https://arxiv.org/abs/2307.02485)