Improvisational Games Proposed as Benchmark for AI Social Intelligence
arXiv paper presents Connections as a benchmark requiring AI agents to demonstrate collaborative social awareness through improvisational wordplay.
arXiv:2604.00284 introduces Connections, an improvisational wordplay game that tests language model agents on knowledge retrieval, summarization, and awareness of other agents' cognitive states. The benchmark requires gauging understanding capabilities of collaborators in a constrained communication environment.
Standard benchmarks such as BIG-bench (arXiv:2206.04615) emphasize individual deductive reasoning but do not evaluate real-time collaborative adaptation or social awareness. Generative agent simulations (arXiv:2304.03442) explore multi-agent interactions yet lack the specific improvisational mechanics that force explicit modeling of partners' knowledge states.
Original coverage of Connections omits explicit links to prior theory-of-mind evaluations in LLMs, which have documented failures in tracking belief states during dynamic exchanges. The game addresses this gap by demanding social intelligence beyond single-agent memory.
Claude: Current agents will need explicit theory-of-mind modules to succeed at Connections, as pure retrieval and reasoning prove insufficient for tracking partner knowledge in real time.
Sources (3)
- [1]Primary Source(https://arxiv.org/abs/2604.00284)
- [2]BIG-bench(https://arxiv.org/abs/2206.04615)
- [3]Generative Agents(https://arxiv.org/abs/2304.03442)