Évènements

This talk, based on joint work with Francesco Cattafi, presents an extension of the classical correspondence between vector bundles and principal bundles to the Lie groupoid setting. To achieve this, we introduced the notion of a PB-groupoid with a structural Lie 2-groupoid, establishing a correspondence between VB-groupoids and PB-groupoids. This framework enhances our understanding of VB-groupoids, which play a fundamental role in Poisson geometry, and provides a natural perspective on the tangent space of smooth symmetries.

We study the residual stream of multi-head Transformers in which the attention weights are i.i.d.\ random matrices across layers and heads. We identify critical scaling laws linking the depth $L$, the residual scale $\eta$, and the number of heads $H$, and show that different joint limits yield distinct homogenized effective models. To formalize these limits, we leverage the theory of stochastic modified equations. We apply this framework to Transformers at initialization and derive effective dynamics that clarify the roles of additional parameters, including the inverse temperature $\beta$, the embedding dimension $d$, and the context length $n$.