IndiaSocialBench is an open benchmark I built to measure how language models respond to emotionally difficult Indian conversations: indirect refusals, family negotiations, honor and shame, grief etiquette, and money between friends, asked in English, Hinglish, Hindi.
Most evaluation work asks whether a model is fluent. This one asks whether it reads the room. Every score links back to the transcript and the judge’s explanation behind it, so the ranking can be argued with rather than taken on faith.
31 models · 50 scored conversations each · 3 languages · 8 dimensions
Indirect speech & face-saving
6.11
Money between friends
6.35
Code-mixing
7.17
Honor & shame
7.18
Ritual & grief etiquette
7.50
Hierarchy & deference
8.07
Family negotiation
8.81
Emotional supportcontrol
8.67
MiniMax M3 · MiniMax: English 8.44 · Hinglish 8.32 · Hindi 7.97 · change −0.47
On all situations, 5 of the top 10 models score lower in Hindi than in English. Across all 31 models the average moves from 7.73 to 7.40 (−0.33).
| Model | English | Hinglish | Hindi | Change |
|---|---|---|---|---|
| Claude Fable 5 | 9.52 | 9.64 | 9.36 | −0.16 |
| Claude Opus 5 | 9.55 | 9.56 | 9.32 | −0.23 |
| Ox Alpha | 9.52 | 9.47 | 9.36 | −0.16 |
| Kimi K3 | 9.20 | 9.66 | 9.27 | +0.07 |
| GPT-5.6 Sol | 8.37 | 8.93 | 9.07 | +0.70 |
| Qwen3.7 Max | 8.54 | 9.10 | 8.71 | +0.17 |
| Claude Sonnet 5 | 8.77 | 8.67 | 8.85 | +0.08 |
| Grok 4.5 | 8.22 | 8.51 | 8.58 | +0.36 |
| MiniMax M3 | 8.44 | 8.32 | 7.97 | −0.47 |
| Gemini 3.6 Flash | 8.26 | 8.31 | 8.06 | −0.20 |
| Average of all 31 | 7.73 | 7.55 | 7.40 | −0.33 |
21 of 31 models score worse when the identical situation arrives in Hindi instead of English. Across the field the average drop is 0.31 points. Same scenarios, same rubric, same judge; only the language changed.
Models average 8.67 on the culture-neutral support control but only 7.31 across the cultural dimensions. They know how to sound caring. They do not yet know how India works.
MiniMax M3 declined ordinary family-life scenarios at 5%. Refusals are reported as their own column rather than folded into the score, so a model cannot look safe by saying nothing.
| Model | Overall | Hindi |
|---|---|---|
| 1Claude Fable 5Anthropic | 9.47 | −0.04 |
| 2Claude Opus 5Anthropic | 9.42 | −0.31 |
| 3Ox AlphaStealth | 9.36 | −0.25 |
| 4Kimi K3Moonshot | 9.34 | −0.02 |
| 5GPT-5.6 SolOpenAI | 8.78 | +0.74 |
| 6Qwen3.7 MaxAlibaba | 8.74 | +0.24 |
| 7Claude Sonnet 5Anthropic | 8.64 | +0.11 |
| 8Grok 4.5xAI | 8.35 | +0.47 |
| 9MiniMax M3MiniMax · partial, 40/50 items | 8.27 | −0.58 |
| 10Gemini 3.6 FlashGoogle | 8.13 | −0.20 |
Every model answers the same items under one blinded rubric with a single uniform judge, and reasoning-capable models run with thinking effort capped low so no model gets a hidden advantage. Overall scores carry a bootstrap 95% confidence interval on the live board.
This is a pilot, not a settled result. It is one judge, a small item count, and a scenario set I wrote myself, so treat the gaps between adjacent models as noise and the field-wide patterns as the finding. GLM-4.7 is excluded from the board (insufficient coverage (provider errors)).
Dataset d7bb3900ca057116 · generated August 26, 2026 · judged by google/gemini-3.1-flash-lite. These figures track the published results directly.