Late-layer masking in Qwen3.6-35B-A3B retains 419/500 outputs after removing 640 of 10,240 experts

The arXiv study applies magnitude-based expert masking to the Qwen3.6-35B-A3B model (256 experts per layer, top-8 routing) across 100-, 300-, and 500-prompt scales on the XLCoST benchmark. Early layers 0-9 and middle layers 10-29 lose substantial output quality under masking, while layers 30-39 and especially 35-39 maintain performance when low-magnitude experts are pruned. Multi-server H100 runs establish that narrow very-late policies mask 640 experts yet retain higher quality than flat policies that mask more experts but degrade results to 150/300 at 300-prompt scale.

Data confirm depth dependence: the 35-39 @ 50% policy yields the best quality-to-expert ratio on the 500-prompt held-out set. Top-k reduction from 8 to 6 experts delivers wall-clock gains on 100-prompt probes without quality loss, though it fails to combine cleanly with aggressive masking. These patterns align with prior routing analyses in Mixtral-8x7B and DeepSeek-V2, where early layers encode critical token routing and late layers handle output refinement.

Operationally, the results support targeted weight surgery on late experts rather than uniform pruning, reducing memory footprint while limiting quality regression. Activation-based scoring and subsequent fine-tuning recovery become feasible next steps. Deployment pipelines can prioritize late-layer policies for inference-time compression before attempting full retraining.