Demystifying Softmax Gating Function in Gaussian Mixture of Experts - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Demystifying Softmax Gating Function in Gaussian Mixture of Experts

Abstract

Understanding parameter estimation of softmax gating Gaussian mixture of experts has remained a long-standing open problem in the literature. It is mainly due to three fundamental theoretical challenges associated with the softmax gating: (i) the identifiability only up to the translation of the parameters; (ii) the intrinsic interaction via partial differential equation between the softmax gating and the expert functions in Gaussian distribution; (iii) the complex dependence between the numerator and denominator of the conditional density of softmax gating Gaussian mixture of experts. We resolve these challenges by proposing novel Vononoi loss functions among parameters and establishing the convergence rates of the maximum likelihood estimator (MLE) for solving parameter estimation in these models. When the number of experts is unknown and over-specified, our findings show a connection between the rate of MLE and a solvability problem of a system of polynomial equations.

Dates and versions

hal-04125060 , version 1 (11-06-2023)

Licence

Identifiers

Cite

Huy Nguyen, Trungtin Nguyen, Nhat Ho. Demystifying Softmax Gating Function in Gaussian Mixture of Experts. Advances in Neural Information Processing Systems, NeurIPS 2023 Spotlight, Acceptance rate 3.6% over 12343 submissions, Dec 2023, New Orleans, United States. pp.1-27. ⟨hal-04125060⟩
61 View
0 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More