The aim of this work is to propose a novel, fast, matrix-free solver for the Poisson problem discretised with High-Order Spectral Element Methods (HO-SEM). This method is based on the use of the Discrete Fourier Transform to reduce the problem to the inversion of the symbol of the operator in frequency space. The solver proposed is endowed with several properties. First, it preserves the efficiency of standard FFT algorithm; then, the matrix storage is minimised; a pseudo-explicit Singular Value Decomposition (SVD) is used for the inversion of the symbols; finally, it can be easily extended to multiple dimensions and non-periodic boundary conditions. In particular, due to the underlying HO-SEM discretisation, the multi-dimensional symbol of the operator can be efficiently computed from the one-dimensional symbol by tensorisation.