We consider in this paper the regularity problem for time-optimal trajectories of a single-input control-affine system on a n-dimensional manifold. We prove that, under generic conditions on the drift and the controlled vector field, any control u associated with an optimal trajectory is smooth out of a countable set of times. More precisely, there exists an integer K, only depending on the dimension n, such that the non-smoothness set of u is made of isolated points, accumulations of isolated points, and so on up to K-th order iterated accumulations.