Intra-subject and inter-subject nonlinear registration based on dense transformations requires the setting of many parameters, mainly for regularization. This task is a major issue, as the global quality of the registration will depend on it. Setting these parameters is, however, very hard, and they may have to be tuned for each patient when processing data acquired by different centers or using different protocols. Thus, we present in this article a method to introduce more coherence in the registration by using fewer degrees of freedom than with a dense registration. This is done by registering the images only on user-defined areas, using a set of affine transformations, which are optimized together in a very efficient manner. Our framework also ensures a smooth and coherent transformation thanks to a new regularization of the affine components. Finally, we ensure an invertible transformation thanks to the Log-Euclidean polyaffine framework. This allows us to get a more robust and very efficient registration method, while obtaining good results as explained below. We performed a qualitative and quantitative evaluation of the obtained results on two applications: first on atlas-based brain segmentation, comparing our results with a dense registration algorithm. Then the second application for which our framework is particularly well suited concerns bone registration in the lower-abdomen area. We obtain in this case a better positioning of the femoral heads than with a dense registration. For both applications, we show a significant improvement in computation time, which is crucial for clinical applications.