Earthquake destroyed many buildings, especially wooden ones, in Japan. Collecting information regarding collapsed buildings during the emergency phase (i.e., 72 h after a disaster) is difficult but essential for rescue activities. This study developed an automatic model to detect collapsed buildings using multiple object tracking (MOT) from aerial videos. Roof damage and pancake collapse are destructions unique to traditional Japanese buildings. Previous studies that detected collapsed buildings using the features of debris or damage failed to discriminate between collapsed and held-up buildings when the buildings have the above Japanese feature. Therefore, this study used the deep learning MOT model to classify collapsed and held-up buildings regardless of debris appearance. The recall and precision of each track of collapsed buildings were 29.1% and 36.7%, respectively, based on cross-validation with the drone video of the 2016 Kumamoto Earthquake. Analysis between the recall and other factors indicated that the aspect ratio, speed, and appearance time of the buildings were significant features for the detection. In the relationship between recall and these factors, we deduce that the recall of track increases to 63.9% if the drone operator films aerial videos effectively. Moreover, this study analyzed effective drone filming and flying way to satisfy some conditions for detection. This result provides recommended filming guides to drone operators for future earthquakes.