Convergent Evolution: How Different Language Models Learn Similar Number Representations
Apr 22, 2026Formal Sciences1 repro
DFDeqing FuTZTianyi ZhouMBMikhail BelkinVSVatsal Sharan+1 more
This paper reveals that different language model architectures (Transformers, RNNs, LSTMs) converge on learning similar periodic number representations with periods at 2, 5, and 10, despite being trained differently. The authors identify a two-tiered hierarchy of these features and explain when models learn geometrically separable representations useful for modular arithmetic.