A method and system for computer-based motion estimation and modeling in a medical image sequence of a patient is disclosed. A medical image sequence of a patient is received. A plurality of frames of the medical image sequence are input to a trained deep neural network. Diffeomorphic deformation fields representing estimated motion between the frames of the medical image sequence input to the trained deep neural network are generated. Future motion, or motion between frames, is predicted from the medical image sequence and at least one predicted next frame is generated using the trained deep neural network. An encoding of the observed motion in the medical image sequence is also generated, which is used for motion classification (e.g., normal or abnormal) or motion synthesis to generate synthetic data.