Yifan Zhang Launches Recurrent Looped Transformer That Carries Latent Computation Across Prompts and Responses
Summary
Yifan Zhang launches the Recurrent Looped Transformer, which preserves hidden states and attention caches across prompt-response boundaries; in 79K-parameter synthetic tests, it reaches 60.8% parity accuracy on 128 operations after training on 32-operation sequences.
Key Points
- Yifan Zhang launches the Recurrent Looped Transformer (RLT), a model that carries recurrent latent computation across prompt and response tokens.
- RLT uses a 48-layer causal encoder and 48-layer recurrent decoder, retaining prior final hidden states and sliding-window attention caches across the prompt-response boundary.
- In 79K-parameter synthetic tests, RLT reaches 60.8% parity accuracy at 128 operations after training on sequences of 32 operations, while five-state-transition accuracy falls to 20.7%.