Recommending students useful and effective learning paths is highly valuable to improve their learning experience. The evaluation of the effectiveness of this recommendation is a challenging task that can be performed online or offline. Online evaluation is highly popular but it relies on actual path recommendations to students, which may have dramatic implications. Offline evaluation relies on static datasets of students' learning activities and simulates paths recommendations. Although easier to run, it is difficult to accurately evaluate offline the effectiveness of a learning path recommendation. To tackle this issue, this work proposes simple offline evaluation measures. We show that they actually allow to characterise and differentiate the algorithms.