GPT-Based LLM Recovers 88.0% of Vignette Items but Only 33.3% of Safety Concerns in 6-Clinician Pilot

The arXiv paper introduces InterviewPlayground, a memory-augmented simulator paired with expert vignettes, to enable repeatable comparison of open-ended psychiatric intake systems. Six clinicians conducted interviews against the same simulated patients; the LLM outperformed on item recall but produced more unsupported clinical inferences and documented safety signals in only one-third of cases where clinicians documented them in two-thirds. These deltas expose the core QA gap: higher surface coverage does not equal clinical fidelity.

The metrics directly test three stated requirements for health-system deployment: cross-style comparability, low clinician burden, and relevance to safety and diagnostic standards. The 56.8% ungrounded-inference rate and 33.3% safety-characterization rate indicate that current LLM intake systems risk both over-diagnosis and under-documentation of risk, patterns already observed in broader hallucination literature on medical LLMs.

Operationally, the platform supplies a repeatable test harness that health systems can run before any production rollout, shifting evaluation from vendor claims to measurable vignette recovery, inference grounding, and safety documentation. Future iterations can add longitudinal vignette sets and regulatory-grade logging to meet emerging FDA and Joint Commission expectations for AI diagnostic tools.

Deployment will require health systems to define minimum thresholds on each axis before allowing LLM intake into live workflows.