We consider the inverse scattering problem for a cavity that is bounded by a penetrable anisotropic inhomogeneous medium of compact support and seek to determine the shape of the cavity from internal measurements on a curve or surface inside the cavity. We derive a factorization method which provides a rigorous characterization of the support of the cavity in terms of the range of an operator which is computable from the measured data. The support of the cavity is determined without a-priori knowledge of the constitutive parameters of the surrounding anisotropic medium provided they satisfy appropriate physical as well as mathematical assumptions imposed by our analysis. Numerical examples are given showing the viability of our method.