Systems and methods for performing a medical imaging analysis task using a machine learning based motion model are provided. One or more medical images of an anatomical structure are received. One or more feature vectors are determined. The one or more feature vectors are mapped to one or more motion vectors using the machine learning based motion model. One or more deformation fields representing motion of the anatomical structure are determined based on the one or more motion vectors and at least one of the one or more medical images. A medical imaging analysis task is performed using the one or more deformation fields.