hal-03829400
https://hal.science/hal-03829400
https://hal.science/hal-03829400v2/document
https://hal.science/hal-03829400v2/file/learning_features.pdf
arxiv:2211.08771
On the symmetries in the dynamics of wide two-layer neural networks
Hajjar, Karl
Chizat, Lenaic
[INFO.INFO-LG] Computer Science [cs]/Machine Learning [cs.LG]
[STAT.ML] Statistics [stat]/Machine Learning [stat.ML]
UNDEFINED
Neural NetworksNN
Infinite-width limit
Gradient Methods
We consider the idealized setting of gradient flow on the population risk for infinitely wide two-layer ReLU neural networks (without bias), and study the effect of symmetries on the learned parameters and predictors. We first describe a general class of symmetries which, when satisfied by the target function $f^*$ and the input distribution, are preserved by the dynamics. We then study more specific cases. When $f^*$ is odd, we show that the dynamics of the predictor reduces to that of a (non-linearly parameterized) linear predictor, and its exponential convergence can be guaranteed. When $f^*$ has a low-dimensional structure, we prove that the gradient flow PDE reduces to a lower-dimensional PDE. Furthermore, we present informal and numerical arguments that suggest that the input neurons align with the lower-dimensional structure of the problem.
2022-10-30
2022-11-24
en