Deep learning methods for solving partial differential equations (PDEs) have recently gained importance thanks to PINNs (Physics Informed Neural Net- works) which can be used to approximate the solution to the problem by avoiding the often delicate and costly discretization step of standard meth- ods such as finite differences or finite elements via the minimization of a cost function. We are interested in the solution of an acoustic wave equation for which we propose a new cost function by considering an ultra-weak variational formulation of the problem to solve. This leads to a new variant of PINNs that we call ultra-weak PINNs (UW-PINN). We use the ultra-weak Trefftz formulation, which is available when solutions of the underlying PDE are explicitly known for simple models. This ultra-weak approach avoids the use of automatic differentiation to minimize the cost function and provides more accurate results. In this work, we present applications of the UW-PINN framework in the context of the 1D acoustic wave equation in multilayered media. We also highlight the advantages of transfer learning in the context of UW-PINN as it significantly improves the learning speed of such a method when efficient approximations are available for similar problems.