This paper presents a physically-based algorithm to interactively simulate and control the motion of soft robots interacting with their environment. We use the Finite Element Method (FEM) to simulate the non-linear deformation of the soft structure, its actuators, and surroundings, and propose a control method relying on a quadratic optimization to find the inverse of the model. The novelty of this work is that the deformations due to contacts, including self-collisions, are taken into account in the optimization process. We propose a dedicated and efficient solver to handle the linear complementarity constraints introduced by the contacts. Thus, the method allows interactive transfer of the motion of soft robots from their task space to their actuator space while interacting with their surrounding. The method is generic and tested on several numerical examples and on a real cable-driven soft robot.