We investigate the influence of thermal haptic feedback on stress during a cognitive task in virtual reality. We hypothesized that cool feedback would help reduce stress in such a task where users are actively engaged. We designed a haptic system using Peltier cells to deliver thermal feedback to the left and right trapezius muscles. A user study was conducted on 36 participants to investigate the influence of different temperatures (cool, warm, neutral) on users' stress during mental arithmetic tasks. Results show that the impact of the thermal feedback depends on the participant's temperature preference. Interestingly, a subset of participants (36%) felt less stressed with cool feedback than with neutral feedback, but had similar performance levels, and expressed a preference for the cool condition. Emotional arousal also tended to be lower with cool feedback for these participants. This suggests that cool thermal feedback has the potential to induce relaxation in cognitive tasks. Taken together, our results pave the way for further experimentation on the influence of thermal feedback.