Executive summary
The clearest result is a regime split. At the 340M / 20B scale, SFDA shows a consistent short-context generalization advantage despite slightly higher late-training cross-entropy: it improves the five-task common-sense macro average by 0.64 percentage points and raises the unweighted mean across the three 1K RULER NIAH variants from 42.1% to 66.5% (+24.4 pp). KDA remains stronger on most extraction-style recall tasks and on longer 2K–4K exact-retention settings.
LAMBADA accuracy and perplexity, WikiText-103 perplexity, and PG-19 perplexity.
Three wins, one tie, and one 0.07-pp near-tie across five zero-shot tasks.
SFDA leads the three-task NIAH mean by 24.4 pp, driven by NIAH2 and NIAH3.
Agreement and reflexives improve; NPI licensing remains a clear counterexample.
The downstream advantage is not explained by a lower SFDA training loss.
FDA/SWDE extraction and most 2K–4K NIAH settings favor KDA.
1. Highlighted held-out language-model benchmarks
These evaluations test generalization beyond the training objective. LAMBADA emphasizes context-sensitive last-word prediction; WikiText-103 measures perplexity on encyclopedic text; PG-19 evaluates long-form book language modeling. Accuracy is higher-is-better and perplexity is lower-is-better.
| Benchmark | SFDA | KDA | SFDA advantage |
|---|---|---|---|
| LAMBADA accuracy ↑Context-sensitive exact prediction | 37.78% | 36.02% | +1.77 pp |
| LAMBADA perplexity ↓Token-level likelihood on LAMBADA | 23.87 | 24.33 | −1.86% |
| WikiText-103 perplexity ↓Held-out encyclopedic language modeling | 24.97 | 25.40 | −1.70% |
| PG-19 perplexity ↓Long-form book language modeling | 116.79 | 118.13 | −1.14% |
The supplied benchmark summary did not identify the exact checkpoint/scale associated with these four values. The numbers are reproduced verbatim; the final external archive should attach the corresponding checkpoint ID and evaluation command.
2. 100M-class model trained on 5B tokens
This experiment asks whether SFDA changes basic optimization behavior at a small but meaningful pretraining scale. Both models see the same total tokens per optimizer update and the same training recipe; microbatch and accumulation are adjusted to fit memory.
3. 340M-class model trained on 20B tokens
This is the larger scaling check. The experiment retains matched data, seed, optimizer recipe, total tokens, and tokens per optimizer update. The runs use different GPU generations and microbatch decompositions, so their training throughput is not directly comparable; loss and downstream quality remain comparable under the matched recipe.
4. LM-SynEval: controlled syntactic generalization
LM-SynEval aggregates 72 official sentence-level zero-shot tasks under matched device, batch, and precision conditions. It probes whether the models assign higher probability to grammatically licensed continuations. The result is not a generic language-model score: it isolates specific syntactic dependencies.
| Metric | SFDA | KDA | SFDA − KDA |
|---|---|---|---|
| LM-SynEval overall | 94.39% | 93.10% | +1.29 pp |
| Subject–verb agreement | 96.57% | 94.38% | +2.19 pp |
| Reflexive anaphora | 88.74% | 87.60% | +1.14 pp |
| Negative-polarity-item licensing | 86.71% | 91.29% | −4.59 pp |
Task composition
| Category | Sentence-level tasks | Share |
|---|---|---|
| Subject–verb agreement | 54 | 75.0% |
| Reflexive anaphora | 10 | 13.9% |
| Negative polarity items | 8 | 11.1% |
| Total | 72 | 100% |
5. Broader evaluation context
The highlighted results are most informative when read together with the complete common-sense and recall suite. The tables below prevent a selective “all metrics improve” interpretation.
Common-sense reasoning
| Scale / task | Low-rank SFDA | KDA | Difference |
|---|---|---|---|
| 100M: ARC-Challenge | 18.34% | 19.03% | −0.69 pp |
| 100M: HellaSwag | 28.35% | 28.35% | 0.00 pp |
| 100M: PIQA | 60.56% | 60.34% | +0.22 pp |
| 100M macro average | 35.75% | 35.91% | −0.16 pp |
| 340M: ARC-Challenge | 26.79% | 25.43% | +1.36 pp |
| 340M: ARC-Easy | 61.20% | 60.35% | +0.85 pp |
| 340M: PIQA | 68.77% | 67.74% | +1.03 pp |
| 340M: HellaSwag | 35.62% | 35.69% | −0.07 pp |
| 340M: OpenBookQA | 25.40% | 25.40% | 0.00 pp |
| 340M macro average | 43.56% | 42.92% | +0.64 pp |
Recall-intensive evaluation
| Setting | Low-rank SFDA | KDA | Reading |
|---|---|---|---|
| 100M FDA-512 | 3.72% | 4.18% | KDA +0.46 pp |
| 100M SWDE-512 | 9.47% | 11.53% | KDA +2.06 pp |
| 100M NIAH1-2K | 2.00% | 4.20% | KDA +2.20 pp |
| 340M FDA-512 | 24.34% | 24.16% | SFDA +0.18 pp |
| 340M FDA-1K / 2K | 11.81% / 7.27% | 14.53% / 10.90% | KDA lead grows with length |
| 340M SWDE-512 / 1K / 2K | 30.37 / 14.81 / 11.72% | 33.74 / 20.34 / 15.93% | KDA leads at all lengths |
| 340M NIAH1-1K | 100.0% | 100.0% | Tie |
| 340M NIAH2-1K | 84.4% | 22.4% | SFDA +62.0 pp |
| 340M NIAH3-1K | 15.0% | 3.8% | SFDA +11.2 pp |
| 340M three-task NIAH mean at 1K | 66.5% | 42.1% | SFDA +24.4 pp |
| 340M NIAH at 2K–4K | KDA is generally stronger, including NIAH1-4K: 98.8% vs. 37.2%. | The 1K advantage does not persist | |
6. Final reported configurations
The report intentionally retains only canonical KDA and the final low-rank SFDA. Common-sense results use raw accuracy only; interrupted, semantically invalid, or non-mainline variants are excluded.
100M / 5B configuration
| Actual parameters | KDA 83.60M SFDA 85.80M |
| Backbone | 12 layers, hidden 640, 10 heads |
| Head dimension | 64 real channels; SFDA uses 32 complex pairs |
| FFN / vocabulary | Ratio 4; 32K vocabulary; tied embeddings |
| Short convolution | Enabled, kernel size 4 |
| Q/K normalization | L2 normalization enabled |
| KDA α projection | 640 → 64 → 640 |
| SFDA α projection | 640 → 64 → 320, pair-tied |
| SFDA phase | θmax=π; zero initialization; logit scale 0.125 |
| Phase optimizer | 0.1× main learning rate |
| SFDA chunk size | 32 |
340M / 20B configuration
| Actual parameters | KDA 399.19M SFDA 408.61M |
| Backbone | 24 layers, hidden 1024, 4 heads |
| Head dimension | 256 real channels; SFDA uses 128 complex pairs |
| FFN / vocabulary | Ratio 4; 32K vocabulary; untied embeddings |
| Short convolution | Disabled for both models |
| KDA α projection | 1024 → 256 → 1024 |
| SFDA α projection | 1024 → 256 → 512, pair-tied |
| SFDA phase | θmax=π; zero initialization; logit scale 0.125 |
| SFDA chunk size | 16 |
| Training sequence | 4096 tokens |
Shared 100M / 5B training recipe
- Sequence length 2048; 9,537 optimizer steps; 5,000,134,656 total tokens.
- 524,288 tokens per optimizer update.
- Fused AdamW; learning rate 4×10−4; β=(0.9,0.95); ε=10−15; weight decay 0.1.
- 477-step warmup, cosine decay to 0.1× initial LR; gradient clipping 1.0.
- BF16 parameters with FP32 reductions;
torch.compile=true; no activation checkpointing or offload. - KDA microbatch/GA: 128 × 2048 × 2; SFDA: 64 × 2048 × 4.
Shared 340M / 20B training recipe
- Sequence length 4096; 38,147 optimizer steps; 20,000,014,336 total tokens.
- 524,288 tokens per optimizer update.
- Fused AdamW with the same LR, betas, epsilon, weight decay, clipping, and cosine schedule as the 100M run.
- 1,908 warmup steps.
- KDA: 4 A800 GPUs, microbatch 4, gradient accumulation 8.
- SFDA: 2 B200 GPUs, microbatch 32, gradient accumulation 2.
- Because hardware and microbatch decomposition differ, throughput is not compared from these two training runs.
7. Interpretation, limitations, and next tests
- SFDA trains stably at 5B and 20B-token scale with loss close to KDA.
- The late-training loss gap narrows at the larger scale.
- SFDA improves all four highlighted held-out LM metrics.
- At the 340M-class scale, SFDA shows a consistent short-context reasoning advantage: three common-sense wins, one tie, one near-tie, and a +0.64-pp macro improvement.
- Across the three 1K NIAH variants, SFDA raises the unweighted mean from 42.1% to 66.5% (+24.4 pp), indicating a pronounced advantage on harder short-context structured-binding tasks.
- LM-SynEval indicates a meaningful shift in syntactic inductive bias.
- Uniform superiority over KDA.
- A uniform advantage across all short-context recall: FDA/SWDE extraction does not show the same pattern as 1K NIAH binding.
- A stable exact-recall advantage at longer context lengths.
- Attributing all downstream gains to phase alone: SFDA has 2.36–2.64% more parameters.
- Statistical significance: current comparisons use one training seed.
- Direct absolute comparison across evaluation protocols with different cache, batch, or generation settings.
Highest-value next experiments
- Equal-parameter control: match KDA and SFDA parameter counts.
- Multiple seeds: at least three seeds for the 100M run and a targeted repeat at 340M.
- Transport-residual diagnostic: compare the prediction error before the delta update under phase transport versus decay-only transport.
- NPI error analysis: identify why SFDA loses 4.59 pp despite gains on agreement and reflexives.
- Regime-split analysis: explain why phase transport helps 1K NIAH2/NIAH3 binding but not FDA/SWDE extraction or longer 2K–4K retention.
- Recall redesign: test phase frequencies, bounded decay, chunk size, and hybrid global-attention ratios on the longer-context failures.
8. References and project links
- Semidirect Fourier Delta Attention
- Kimi Linear / Kimi Delta Attention
- LM Evaluation Harness
- BASED: recall–throughput evaluation
- RULER long-context benchmark
- FineWeb-Edu
Canonical source scope: only final low-rank SFDA and canonical KDA are retained; common-sense evaluation reports raw accuracy; interrupted, semantically invalid, and non-mainline experiments are excluded.