Oyster-II Zero-RL reaches Qwen3-Max safety parity on 14B scale

The arXiv 2607.02914 paper replaces Oyster-I's SFT stage with a Zero-RL objective plus staged reward modeling. This directly targets two documented SFT failure modes: OOD safety collapse and safety CoT over-generalization that penalizes benign queries. The method trains a 14B base model without supervised refusal data, using only outcome-based safety and helpfulness rewards.

Benchmarks show Oyster-II exceeding both Qwen3-14B and Oyster-I on safety suites while retaining or improving helpfulness scores. The gains are largest on out-of-distribution adversarial sets, indicating the RL signal produces broader refusal boundaries than token-level SFT. No new model weights are released; results rest on internal evals comparable to those used for Qwen3 series.

The result isolates RL as the variable that scales constructive safety past superficial RLHF refusal patterns. It demonstrates that reward shaping on intent resolution rather than surface refusal can close the generalization gap without the helpfulness tax observed in pure SFT runs. Operational implication is that future agent stacks can insert this RL stage after pre-training instead of relying on post-hoc alignment patches.

Next steps include public release of the reward model and staged training code; absence of those artifacts limits independent verification of the reported deltas.