We describe a hierarchy of computational models of the human body operating at the geometrical, physical and physiological levels. Those models can be coupled with medical images which play a crucial role in the diagnosis, planning, control and follow-up of therapy. In this paper, we discuss the issue of building patient-specific physical and physiological models from macroscopic observations extracted from medical images. We illustrate the topic of model personalization with concrete examples in brain shift modeling, hepatic surgery simulation, cardiac and tumor growth modeling. We conclude this article with scientific perspectives.