The Speedup Test - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Reports (Technical Report) Year : 2010

The Speedup Test


Numerous code optimisation methods are usually experimented by doing multiple observations of the initial and the optimised executions times in order to declare a speedup. Even with fixed input and execution environment, programs executions times vary in general. So hence different kinds of speedups may be reported: the speedup of the average execution time, the speedup of the minimal execution time, the speedup of the median, etc. Many published speedups in the literature are observations of a set of experiments. In order to improve the reproducibility of the experimental results, this technical report presents a rigorous statistical methodology regarding program performance analysis. We rely on well known statistical tests (Shapiro-wilk's test, Fisher's F-test, Student's t-test, Kolmogorov-Smirnov's test, Wilcoxon-Mann-Whitney's test) to study if the observed speedups are statistically significant or not. By fixing $0<\alpha<1$ a desired risk level, we are able to analyse the statistical significance of the average execution time as well as the median. We can also check if $\prob{X>Y}>\frac{1}{2}$, the probability that an individual execution of the optimised code is faster than the individual execution of the initial code. Our methodology defines a consistent improvement compared to the usual performance analysis method in high performance computing as in \cite{Jain:1991:ACS,lilja:book}. We explain in each situation what are the hypothesis that must be checked to declare a correct risk level for the statistics. The Speedup-Test protocol certifying the observed speedups with rigorous statistics is implemented and distributed as an open source tool based on R software.
Fichier principal
Vignette du fichier
SpeedupTestDocument.pdf (587.91 Ko) Télécharger le fichier
SpeedupTestSlides.pdf (8.28 Mo) Télécharger le fichier
SpeedupTestTool.tar.bz2 (607.91 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Format : Other
Format : Other

Dates and versions

inria-00443839 , version 1 (07-01-2010)
inria-00443839 , version 2 (26-01-2010)


  • HAL Id : inria-00443839 , version 2


Sid Touati, Julien Worms, Sébastien Briais. The Speedup Test. [Technical Report] Université Versailles Saint-Quentin-En-Yvelines (UVSQ). 2010, pp.38. ⟨inria-00443839v2⟩
1495 View
1221 Download


Gmail Facebook Twitter LinkedIn More