A new variant of the IPDG method is presented, involving carefully constructed weighted averages of the gradient of the approximate solution. The method is shown to be robust even for the most extreme simultaneous local mesh, polynomial degree and diffusion coefficient variation scenarios, without resulting into unreasonably large penalization. The new IPDG method, henceforth termed as robust IPDG (RIPDG), offers typically significantly better error behaviour and conditioning than the standard IPDG method when applied to scenarios with strong mesh/polynomial degree/diffusion local variation, especially when the underlying approximation space does not contain a sufficiently rich conforming subspace. The latter is of particular importance in the context of IPDG methods on polygonal/polyhedral meshes. On the other hand, when using uniform meshes, constant polynomial degree for problems with constant diffusion coefficients the RIPDG method is identical to the classical IPDG. Numerical experiments indicate the favourable performance of the new RIPDG method over the classical version in terms of conditioning and error.