In this paper, we study approximate and exact controllability of the linear difference equation $x(t) = \sum_{j=1}^N A_j x(t - \Lambda_j) + B u(t)$ in $L^2$, with $x(t) \in \mathbb C^d$ and $u(t) \in \mathbb C^m$, using as a basic tool a representation formula for its solution in terms of the initial condition, the control $u$, and some suitable matrix coefficients. When $\Lambda_1, \dotsc, \Lambda_N$ are commensurable, approximate and exact controllability are equivalent and can be characterized by a Kalman criterion. This paper focuses on providing characterizations of approximate and exact controllability without the commensurability assumption. In the case of two-dimensional systems with two delays, we obtain an explicit characterization of approximate and exact controllability in terms of the parameters of the problem. In the general setting, we prove that approximate controllability from zero to constant states is equivalent to approximate controllability in $L^2$. The corresponding result for exact controllability is true at least for two-dimensional systems with two delays.