The authors build a mathematical framework that describes how transformer neural networks behave when you take two key quantities to be very large: the number of input tokens (pieces of text) and the number of attention heads (parallel processing units) in each layer. By also treating the network as having infinitely many layers arranged along a continuous time axis rather than discrete steps, they arrive at a cleaner, more tractable description. Instead of tracking every individual parameter and token separately, the framework works with probability distributions over tokens and over parameters, which is the hallmark of what physicists call a mean-field approach. Two coupled objects evolve together: one distribution describes how the collection of tokens is transformed as information flows through the network's depth, and another describes how the network's attention parameters change during training.
The paper's central technical achievement is proving that this coupled system of equations is well-posed, meaning it has a unique, stable solution that exists for all time. This is non-trivial because the two distributions influence each other simultaneously, creating a complex feedback loop described by equations from the theory of interacting particle systems. The authors also include optional regularization terms, which are common practical tricks used during training to prevent overfitting, and they show these fit naturally into the framework.
Beyond just showing the equations make sense, the paper connects the theory to the practical question of whether training actually finds a good solution. For simpler, single-layer versions of the model, they prove that training converges exponentially fast to the best possible solution, under a geometric condition on the parameter space. For full deep transformers, they prove that once training is close enough to a good solution, it converges at a linear rate, using a condition related to the Neural Tangent Kernel, a standard tool for analyzing overparameterized networks. Together, these results provide one of the most rigorous mathematical accounts of why transformer training works.