Previous papers have proposed to add memory registers to the dynamics of discrete-time linear systems in order to accelerate their convergence. In particular, it has been proved that adding one memory slot per agent allows faster convergence towards average consensus. We here prove that this situation cannot be improved by adding more memory slots, when the knowledge about the self-adjoint linear map to be accelerated reduces to bounds on its extreme eigenvalues.