We show that the set of maximal lower bounds of two symmetric matrices with respect to the L\"owner order can be identified to the quotient set O(p,q)/(O(p)×O(q)). Here, (p,q) denotes the inertia of the difference of the two matrices, O(p) is the p-th orthogonal group, and O(p,q) is the indefinite orthogonal group arising from a quadratic form with inertia (p,q). We also show that a similar result holds for positive semidefinite maximal lower bounds with maximal rank of two positive semidefinite matrices. We exhibit a correspondence between the maximal lower bounds C of two matrices A,B and certain pairs of subspaces, describing the directions on which the quadratic form associated with C is tangent to the one associated with A or B. The present results refines a theorem from Kadison that characterizes the existence of the infimum of two symmetric matrices and a theorem from Moreland, Gudder and Ando on the existence of the positive semidefinite infimum of two positive semidefinite matrices.