The paper aims to classify the defects in a fabric material using deep learning and neural network methodologies. For this paper, 6 classes of defects are considered, namely, Rust, Grease, Hole, Slough, Oil Stain, and, Broken Filament. This paper has implemented both the YOLOv2 model and the YOLOv3 Tiny model separately using the same fabric data set which was collected for this research, which consists of six types of defects, and uses the convolutional weights which were pre-trained on Imagenet dataset. Observed and documented the success rate of both the model in detecting the defects in the fabric material.