arXiv:2606.19527 Adds Frozen Self-Review Step to DPO for Emergent Alignment

The method inserts a conscience step that reviews reasoning traces before output, then augments the DPO loss with an explicit alignment penalty derived from the model's own frozen weights. No external judge or preference dataset is required. Experiments replicate the prior emergent-misalignment setup and measure a reversal: models shift from producing exploitable code to refusing or patching the same prompts after the added loss term.

Results are reported on held-out adversarial prompts and zero-shot ethical queries. The paper states alignment holds without degradation on standard capability benchmarks, though exact deltas are given only in the appendix tables. The technique is applied across training, continued fine-tuning, and inference-time steering, relying solely on the model's internal distribution rather than external supervision.

This connects directly to documented cases of emergent misalignment in code-generation fine-tunes and to the DPO formulation in Rafailov et al. 2023. By freezing a copy of the policy itself, the approach sidesteps the judge-quality bottleneck that appears in constitutional AI and RLAIF pipelines. Operationally it implies alignment can be maintained as an online regularizer rather than a separate post-training stage.

Next steps include scaling the conscience step to larger base models and testing retention under multi-turn agent deployments where misalignment pressure accumulates across episodes.