A Close Look at Multi-tenant Parallel CNN Inference for Autonomous Driving - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

A Close Look at Multi-tenant Parallel CNN Inference for Autonomous Driving

Résumé

Convolutional neural networks (CNNs) are widely used in vision-based autonomous driving, i.e., detecting and localizing objects captured in live video streams. Although CNNs demonstrate the state-of-the-art detection accuracy, processing multiple video streams using such models in real-time imposes a serious challenge to the on-car computing systems. The lack of optimized system support, for example, could lead to a significant frame loss due to the high processing latency, which is unacceptable for safety-critical applications. To alleviate this problem, several optimization strategies such as batching, GPU parallelism, and data transfer modes between CPU/GPU have been proposed, in addition to a variety of deep learning frameworks and GPUs. It is, however, unclear how these techniques interact with each other, which particular combination performs better, and under what settings. In this paper, we set out to answer these questions. We design and develop a Multi-Tenant Parallel CNN Inference Framework, MPInfer, to carefully evaluate the performance of various parallel execution modes with different data transfer modes between CPU/GPU and GPU platforms. We find that on more powerful GPUs such as GTX 1660, it achieves the best performance when we adopt parallelism across CUDA contexts enhanced by NVIDIA Multi-Process Service (MPS), with 147.06 FPS throughput and 14.50 ms latency. Meanwhile, on embedded GPUs such as Jetson AGX Xavier, pipelining is a better choice, with 46.63 FPS throughput and 35.09 ms latency.
Fichier principal
Vignette du fichier
511910_1_En_8_Chapter.pdf (1.05 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03768759 , version 1 (04-09-2022)

Licence

Paternité

Identifiants

Citer

Yitong Huang, Yu Zhang, Boyuan Feng, Xing Guo, Yanyong Zhang, et al.. A Close Look at Multi-tenant Parallel CNN Inference for Autonomous Driving. 17th IFIP International Conference on Network and Parallel Computing (NPC), Sep 2020, Zhengzhou, China. pp.92-104, ⟨10.1007/978-3-030-79478-1_8⟩. ⟨hal-03768759⟩
19 Consultations
4 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More