One important problem in diffusion MRI (dMRI) is to recover the diffusion weighted signal from only a limited number of samples in q-space. An ideal framework for solving this problem is Compressed Sensing (CS), which takes advantage of the signal's sparseness or compressibility, allowing the entire signal to be reconstructed from relatively few measurements. CS theory requires a suitable dictionary that sparsely represents the signal. To date in dMRI there are two kinds of Dictionary Learning (DL) methods: 1) discrete representation based DL (DR-DL), and 2) continuous representation based DL (CR-DL). Due to the discretization in q-space, DR-DL suffers from the numerical errors in interpolation and regridding. By considering a continuous representation using Spherical Polar Fourier (SPF) basis, this paper proposes a novel CR-DL based Spherical Polar Fourier Imaging, called DL-SPFI, to recover the diffusion signal as well as the Ensemble Average Propagator (EAP) in continuous 3D space with closed form. DL-SPFI learns an optimal dictionary from the space of Gaussian diffusion signals. Then the learned dictionary is adaptively applied for different voxels in a weighted LASSO framework to robustly recover the di ffusion signal and the EAP. Compared with the start-of-the-art CR-DL method by Merlet et al. and DRDL by Bilgic et al., DL-SPFI has several advantages. First, the learned dictionary, which is proved to be optimal in the space of Gaussian diffusion signal, can be applied adaptively for different voxels. To our knowledge, this is the first work to learn a voxel-adaptive dictionary. The importance of this will be shown theoretically and empirically in the context of EAP estimation. Second, based on the theoretical analysis of SPF basis, we devise an efficient learning process in a small subspace of SPF coefficients, not directly in q-space as done by Merlet et al.. Third, DL-SPFI also devises different regularization for different atoms in the learned dictionary for robust estimation, by considering the structural prior in the space of signal exemplars. We evaluate DL-SPFI in comparison to L1-norm regularized SPFI (L1-SPFI) with fixed SPF basis, and the DR-DL by Bilgic et al. The experiments on synthetic data and real data demonstrate that the learned dictionary is sparser than SPF basis and yields lower reconstruction error than Bilgic's method, even though only simple synthetic Gaussian signals were used for training in DL-SPFI in contrast to real data used by Bilgic et al.