Railway safety faces a critical challenge from overheated axle bearings, commonly known as hot boxes. Early detection is, therefore, of utmost importance. Departing from the complexity and resource-heavy nature of current hot box detectors, a new and affordable thermal vision-based wayside monitoring method is proposed. It combines panoramic thermal images with deep learning (DL)-based object detection algorithm. Panoramas were created, from thermal image sequences collected, by cooled and uncooled cameras, during field experiments on both freight and passenger trains. The results yielded promising perspectives, notably in maintaining high safety standards, at a more budget-friendly cost.