Given that heart disease can cause abnormal motion dynamics over the cardiac cycle, which can then affect cardiac function, understanding and quantifying cardiac motion can provide insight for clinicians to aid in diagnosis, therapy planning, as well as to determine the prognosis for a given patient. The goal of this paper is to extract population-specific cardiac motion patterns from 3D displacements in order to firstly identify the mean motion behaviour in a population and secondly to describe pathology-specific motion patterns in terms of the spatial and temporal aspects of the motion. Since there are common motion patterns observed in patients suffering from the same condition, extracting these patterns can lead towards a better understanding of a disease. Quantifying cardiac motion at a population level is not a simple task since images can vary widely in terms of image quality, size, resolution and pose. To overcome this, we analyse the parameters obtained from a cardiac-specific Polyaffine motion tracking algorithm, which are aligned both spatially and temporally to a common reference space. Once all parameters are aligned, different subjects and different populations can be compared and analysed in the space of Polyaffine transformations by projecting the transformations to a reduced-order subspace in which dominant motion patterns in each population can be extracted and analysed. Using tensor decomposition allows the spatial and temporal aspects to be decoupled in order to study the different components individually. The proposed method was validated on healthy volunteers and Tetralogy of Fallot patients according to known spatial and temporal behaviour for each population. A key advantage of the proposed method is the ability to regenerate motion sequences from the respective models, thus the models can be visualised in terms of the full motion, which allows for better understanding of the motion dynamics of different populations.