The Pitfalls of Benchmarking with Applications - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2010

The Pitfalls of Benchmarking with Applications

Erven Rohou
Thierry Lafage
  • Function : Author
  • PersonId : 869080


Application benchmarking is a widely trusted method of performance evaluation. Compiler developers rely on them to assess the correctness and performance of their optimizations; computer vendors use them to compare their respective machines; processor architects run them to tune innovative features, and — to a lesser extent — to validate their correctness. Benchmarks must reflect actual workloads of interest, and return a synthetic measure of “performance”. Often, benchmarks are simply a collection of real-world applications run as black boxes. We identify a number of pitfalls that derive from using applications as benchmarks, and we illustrate them with a popular, freely available, benchmark suite. In particular, we advocate the fact that correctness should be defined by an expert of the application domain, and the test should be integrated in the benchmark.
Fichier principal
Vignette du fichier
4-mobs6-rohou.pdf (235.54 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

inria-00492997 , version 1 (17-06-2010)


  • HAL Id : inria-00492997 , version 1


Erven Rohou, Thierry Lafage. The Pitfalls of Benchmarking with Applications. MoBS 2010 - Sixth Annual Workshop on Modeling, Benchmarking and Simulation, Jun 2010, Saint Malo, France. ⟨inria-00492997⟩
179 View
214 Download


Gmail Facebook X LinkedIn More