Capacity planning is an important production control function that significantly influences firm performance. Especially, in the short term, we face a dynamically changing system which calls for an adaptive capacity planning system that reacts based on the current state of the shop floor. Thus, this paper analyzes the performance of a reinforcement learning (RL) algorithm for overtime planning for a make-to-order job shop. We compare the performance of the RL algorithm to mechanisms that set overtime-hours statically or randomly over time. Performance is measured in total costs which consist of overtime, holding and backorder costs. The results show that our tested benchmarks can be outperformed by the RL algorithm, where the major savings were achieved due to less needed overtime.