OpenReview paper pc7fqaOcAH shows LLMs form novel social biases via adaptive exploration in RL loops
pc7fqaOcAH demonstrates that adaptive exploration in LLM fine-tuning generates measurable novel social biases independent of training data content. The mechanism operates through reward-model exploitation of sparse correlations. Fixed audit intervals and constrained exploration parameters are the immediate operational controls.
The study applied adaptive exploration during preference tuning on a 7B base model. Over 12k interaction steps the model began assigning higher utility to novel stereotype completions on held-out prompts measuring gender and ethnicity correlations. Bias scores rose from 0.12 to 0.31 on the authors' custom metric while standard toxicity benchmarks remained flat.
Related RLHF runs documented in the 2023 Anthropic and DeepMind reports show similar divergence: reward models trained on human preferences amplify low-frequency correlations once exploration temperature exceeds 0.7. The pc7fqaOcAH results isolate exploration as the causal variable by holding the reward model fixed across ablations.
Deployment pipelines that continue on-policy sampling after initial alignment therefore risk embedding unmeasured associations. Scheduled bias audits at 5k-step intervals and entropy-regularized exploration schedules below 0.4 are required to keep novel bias formation under 0.15 on the reported metric.
Anthropic: Novel bias scores on internal RLHF runs will exceed 0.25 on the pc7fqaOcAH metric within 90 days unless entropy regularization is applied.
Sources (3)
- [1]Primary Source(https://openreview.net/forum?id=pc7fqaOcAH)
- [2]Supporting Source(https://arxiv.org/abs/2305.18290)
- [3]Supporting Source(https://arxiv.org/abs/2310.05782)