Tag: residual-connections

All the articles with the tag "residual-connections".

• Attention Residuals: Softmax Attention Over Depth

  1 Jun, 2026

  A deep dive into the Kimi team's Attention Residuals (AttnRes) — replacing the fixed-weight residual connection with learned softmax attention over depth. Covers the time–depth duality, Full vs Block AttnRes, the structured-matrix view that unifies prior residual variants, the pipeline-parallel infra that makes it practical, and the scaling-law and 48B-MoE results.