The contributions of this book demonstrate a wide variety of image synthesis and simulation methods, from parametric modeling to deep learning, and their application to diverse tasks such as image enhancement or data augmentation. The ultimate goal when developing methods is to design a simulation system that can produce realistic anatomical or biological images for diverse acquisition conditions and that is fully controllable, accurate, robust, simple to use, fast and easily accessible to all. This would ideally lead to simulated/augmented data of high quality, high variability and high fidelity (both spatially and in time). However, several challenges remain. This chapter will highlight current limitations and identify possible future research directions. Methods used for the processing and analysis of medical imaging data often come from the computer vision field. However, medical images have different characteristics than natural images and therefore standard computer vision methods cannot be directly used for image synthesis. This is for example the case of deep learning methods as many imaging modalities are intrinsically 3D, meaning that networks built for 2D images must be redesigned, and the amount of data samples available to train networks is far below that of natural images. Without sufficient training samples, the results are thus suboptimal. This is accentuated by the fact that training still often requires paired data (i.e. pairs of images from different modalities but also pairs of images and annotations), which are difficult to gather, for instance because of the invasiveness of a modality or the difficulty to obtain annotations. The development N. Burgos and D. Svoboda (Eds.), Biomedical Image Synthesis and Simulation, Elsevier