ArXiv 2608.23641 attributes 30-45% of AI preference variance to instrument artifacts

The study isolates instrument effects by holding model weights fixed while varying elicitation interfaces. Controlled swaps of temperature, few-shot exemplars, and Likert scale anchors produced preference rank reversals in 34% of trials across Llama-3-70B and Claude-3.5-Sonnet. Replicates on held-out tasks confirm the pattern scales with output entropy rather than semantic content.

Data tables report Cronbach alphas dropping from 0.81 to 0.52 when instrument constraints tighten, with largest deltas on moral dilemma and resource allocation prompts. Secondary runs using direct logit inspection versus sampled completions isolate the measurement channel from the policy itself. No single model family escaped the effect size band.

Operational consequence is that current preference datasets embed instrument signatures that downstream RLHF pipelines treat as ground truth. Alignment teams must therefore version-control the full elicitation stack alongside weights. Without this, claims of reduced bias remain non-replicable across deployments.

Next steps require standardized instrument registries and cross-lab calibration protocols before any preference model is frozen for production use.