Beyond Memory Gating:
Phase as a Missing Axis in Linear-Attention State Transitions

Connection to State Tracking in Transformers

A recent DeepMind paper The Topological Trouble With Transformers , argues that purely feedforward Transformer architectures face a structural difficulty with dynamic state tracking. The core issue is not simply context length: the model may retrieve past text, but maintaining an evolving latent state across many updates is a different computational problem.

This perspective is closely aligned with the motivation for SFDA. Current linear-attention and delta-memory systems already introduce recurrence:

However, in KDA-style systems the transition operator is still largely contractive:

This gives the model a mechanism for writing, erasing, and forgetting, but it does not naturally provide oscillatory or reversible state evolution. SFDA asks whether the next bottleneck is not recurrence itself, but the algebra of the recurrent transition.

Under this view, SFDA is one possible response to the state-tracking problem: keep the efficiency of KDA-style recurrent memory, but enrich the internal transition dynamics so the state can rotate, cycle, and represent structured automata rather than only decay.

From Automata Representations to Associative Memory

SFDA did not begin as a linear-attention modification. The original research question was how Transformers can learn structured sequential computations such as finite automata. The earlier Automata--Transformer project studied whether a long sequence of state transitions must be represented by explicit state enumeration, or whether algebraic structure can provide a compressed representation.

A finite-state transition sequence can be viewed as a product in a transition monoid:

If this composition admits a low-dimensional representation \(\rho(m_2m_1)=\rho(m_2)\rho(m_1)\), then the computation can be carried by a compact representation rather than a growing history. Fourier representations of cyclic groups provide the simplest example:

This raised the next question: once a compact representation exists, how should a neural model maintain and update it online? SFDA is a continuation of this question: the state is no longer only a hidden representation, but an associative memory that predicts and corrects.

SFDA as Transported Associative Memory

The current formulation is more precisely understood as a transported associative memory rather than simply "KDA with phase". KDA already introduced a matrix-valued memory whose read operation is \[ \widehat v_t=S_t^\top k_t . \] The delta update performs online correction of the prediction error.

SFDA inserts one additional operation before this correction:

Therefore phase does not replace associative memory. It changes the coordinate system in which the memory is queried and corrected. The ordering matters:

This is different from adding a KDA branch and a complex state-space branch independently, because the phase transform changes the prediction residual that drives the erase-write update.

Relation to Kimi Delta Attention, Gated DeltaNet, and Mamba-3

SFDA should be viewed as a neighboring evolution of recent efficient sequence models rather than a replacement for them.

The distinction is that Mamba-style models enrich the evolution of an SSM hidden state, while SFDA evolves a key-value associative predictor. KDA learns the retention geometry; SFDA asks whether the memory coordinate system itself should evolve through structured phase dynamics.

Implementation: From Algebra to GPU Kernels

The theoretical contribution is only one part of SFDA. A practical implementation must preserve the structured recurrence while respecting GPU execution constraints.

The implementation follows the same principle as recent efficient attention systems: mathematical expressivity and hardware efficiency must be designed together. The current implementation includes:

• relative-decay WY/UT factorization without forming unstable \(1/\alpha\) terms;
• chunk-parallel forward computation with Triton kernels;
• custom backward kernels with mixed-precision support;
• recompute-aware memory management for long-context training;
• packed-sequence boundary handling through explicit state resets.

The remaining systems challenge is not the recurrence itself, but matching mature KDA implementations in kernel efficiency: reducing activation materialization, improving Tensor Core utilization, and designing hardware-native phase-memory kernels.