Qwen3.5 4B embeds compiled RISC-V CPU in weights using Gated DeltaNet state as addressable RAM

The implementation overwrites GDN rows with one-hot keys at gate=1 and beta=1, turning the linear RNN update into deterministic read-write memory whose size stays constant regardless of context length. This differs from prior associative-memory results in Schlag et al. ICML 2021 by achieving exact, addressable storage inside an unmodified production model rather than a trained fast-weight network. No gradient steps occur; a circuit compiler directly sets layer weights to implement each CPU instruction as a fixed sequence of matrix operations.

Default Qwen3.5 GDN heads do not preserve exact values across tokens. The modification forces persistent one-hot addressing, enabling the CPU to store its program counter, registers, and an experimental in-model filesystem entirely inside the attention state. The approach sidesteps KV-cache growth that would otherwise make attention-based RAM scale linearly with sequence length.

Operationally this demonstrates that linear-attention state can serve as reliable scratchpad memory when gates and writes are constrained to binary extremes. It also supplies a concrete mechanism for embedding non-tamperable monitors or hidden reasoning traces inside the same RNN state that the model uses for generation, a pattern already explored in the follow-up experiments referenced in the source.

Next steps include measuring cycle counts per instruction against silicon baselines and testing whether the same compilation technique scales to 32-bit or 64-bit RISC-V variants within larger Qwen checkpoints.