In this paper, we investigate a practical framework to compute a 3D shape estimation of multiple objects in real-time from silhouettes in multi-view environments. A popular method called Shape From Silhouette (SFS), computes a 3D shape estimation from binary silhouette masks. This method has several limitations: The acquisition space is limited to the intersection of the camera viewing frusta ; SFS methods reconstruct some ghost objects which do not contain real objects, especially when there are multiple real objects in the scene. In this paper we propose two contributions to overcome these limitations. First, using a new formulation of SFS approach, our system reconstructs objects with no constraints on camera placement and their visibility. Second, a new theoretical approach identifies and removes ghost objects. The reconstructed shapes are more accurate than current silhouette-based approaches. Reconstructed parts are guaranteed to contain real objects. Finally, we present a real-time system that captures multiple and complex objects moving through many camera frusta to demonstrate the application and robustness of our method.