Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. Most sampling procedures are based solely on the sampling of the agent's policy. However, other measures directly available through these algorithms could be used to improve sampling before each policy update. Following this line of thoughts, we propose the use of SAUNA, a method where a transition is rejected from the gradient update if it does not meet a particular criterion, and kept otherwise. This criterion is the fraction of variance explained $\mathcal{V}^{ex}$, a measure of the discrepancy between a model and actual samples. $\mathcal{V}^{ex}$ can be used to evaluate the impact each transition will have on the learning. This criterion refines sampling and improves the policy gradient algorithm. In this paper: (a) We introduce and explore $\mathcal{V}^{ex}$, the selection criterion used to improve the sampling procedure. (b) We conduct experiments across a variety of benchmark environments, including standard continuous control problems. Our results show better performance than if we did not use the $\mathcal{V}^{ex}$ criterion for the policy gradient update. (c) We investigate why $\mathcal{V}^{ex}$ provides a reliable assessment for the selection of samples that will have a positive impact on learning. (d) We show how to interpret this criterion as a dynamic way to adjust the ratio between exploration and exploitation.