If $X\subset \mathbb{P}^n$ is a projective non degenerate variety, the $X$-rank of a point $P\in \mathbb{P}^n$ is defined to be the minimum integer $r$ such that $P$ belongs to the span of $r$ points of $X$. We describe the complete stratification of the fourth secant variety of any Veronese variety $X$ via the $X$-rank. This result has an equivalent translation in terms both of symmetric tensors and homogeneous polynomials. It allows to classify all the possible integers $r$ that can occur in the minimal decomposition of either a symmetric tensor or a homogeneous polynomial of $X$-border rank $4$ (i.e. contained in the fourth secant variety) as a linear combination of either completely decomposable tensors or powers of linear forms respectively.