Surround the Nonlinearity: Inserting Foldable Convolutional Autoencoders to Reduce Activation Footprint - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Surround the Nonlinearity: Inserting Foldable Convolutional Autoencoders to Reduce Activation Footprint

Résumé

Modern deep learning architectures, while highly successful, are characterized by substantial computational and memory demands due to their large number of parameters or the storing of activations. That is why it is hard to adapt a neural network to the constraints of hardware, especially at the edge. This paper presents an investigation into a novel approach for activation compression, which we term 'Projection-based compression on channels' or 'ProChan'. Our method involves interposing projection layers into a pretrained network around the nonlinearity, reducing the channel dimensionality through compression operations and then expanding it back. Our module is made to be then totally fused with the convolutions around it, guaranteeing no overhead, and maximum FLOPs reduction. We studied its absorption of the cost of quantization, to combine the two approaches for footprint reduction. Our findings indicate that the projections likely perform an 'adaptive stretching' operation on the feature space, enabling the preservation of essential information when constrained by dimensional limitations. We also perform an ablation study on the different possible strategies for a stable and quick training, and analyse the interactions with different quantization paradigms, namely PACT for activations and posttraining quantization (PTQ) methods for weights.
Fichier principal
Vignette du fichier
ICCV_v_HAL.pdf (1.02 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04400777 , version 1 (17-01-2024)

Licence

Paternité

Identifiants

Citer

Baptiste Rossigneux, Inna Kucher, Vincent Lorrain, Emmanuel Casseau. Surround the Nonlinearity: Inserting Foldable Convolutional Autoencoders to Reduce Activation Footprint. ICCVW 2023 - IEEE/CVF International Conference on Computer Vision Workshops, Oct 2023, Paris, France. pp.1399-1403, ⟨10.1109/ICCVW60793.2023.00152⟩. ⟨hal-04400777⟩
20 Consultations
8 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More