Mixing Low-Precision Formats in Multiply-Accumulate Units for DNN Training - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Mixing Low-Precision Formats in Multiply-Accumulate Units for DNN Training

Abstract

The most compute-intensive stage of deep neural network (DNN) training is matrix multiplication where the multiply-accumulate (MAC) operator is key. To reduce training costs, we consider using low-precision arithmetic for MAC operations. While low-precision training has been investigated in prior work, the focus has been on reducing the number of bits in weights or activations without compromising accuracy. In contrast, the focus in this paper is on implementation details beyond weight or activation width that affect area and accuracy. In particular, we investigate the impact of fixed-versus floating-point representations, multiplier rounding, and floatingpoint exceptional value support. Results suggest that (1) lowprecision floating-point is more area-effective than fixed-point for multiplication, (2) standard IEEE-754 rules for subnormals, NaNs, and intermediate rounding serve little to no value in terms of accuracy but contribute significantly to area, (3) lowprecision MACs require an adaptive loss-scaling step during training to compensate for limited representation range, and (4) fixed-point is more area-effective for accumulation, but the cost of format conversion and downstream logic can swamp the savings. Finally, we note that future work should investigate accumulation structures beyond the MAC level to achieve further gains.
Fichier principal
Vignette du fichier
fpt_2022.pdf (569.78 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03885471 , version 1 (05-12-2022)

Licence

Identifiers

  • HAL Id : hal-03885471 , version 1

Cite

Mariko Tatsumi, Silviu-Ioan Filip, Caroline White, Olivier Sentieys, Guy Lemieux. Mixing Low-Precision Formats in Multiply-Accumulate Units for DNN Training. FPT 2022 - IEEE International Conference on Field Programmable Technology, Dec 2022, Hong Kong, Hong Kong SAR China. pp.1-9. ⟨hal-03885471⟩
77 View
571 Download

Share

Gmail Mastodon Facebook X LinkedIn More