We prove error estimates for the wave equation semi-discretized in space by the hybrid high-order (HHO) method. These estimates lead to optimal convergence rates for smooth solutions. We consider first the second-order formulation in time, for which we establish $H^1$ and $L^2$-error estimates, and the first-order formulation, which has close links with the hybridizable discontinuous Galerkin approach and for which we establish $H^1$-error estimates. Numerical experiments using either the Newmark scheme or diagonally-implicit Runge-Kutta schemes for the time discretization illustrate the theoretical findings and show that the proposed numerical schemes can be used to simulate accurately the propagation of elastic waves in heterogeneous media.