In this work, we propose a high-order multiscale method for an elliptic model problem with rough and possibly highly oscillatory coefficients. Convergence rates of higher order are obtained using the regularity of the right-hand side only. Hence, no restrictive assumptions on the coefficient, the domain, or the exact solution are required. In the spirit of the Localized Orthogonal Decomposition, the method constructs coarse problem-adapted ansatz spaces by solving auxiliary problems on local subdomains. More precisely, our approach is based on the strategy presented by Maier [SIAM J. Numer. Anal. 59(2), 2021]. The unique selling point of the proposed method is an improved localization strategy curing the effect of deteriorating errors with respect to the mesh size when the local subdomains are not large enough. We present a rigorous a priori error analysis and demonstrate the performance of the method in a series of numerical experiments.