Initial Experiments with Duet Benchmarking: Performance Testing Interference in the Cloud

Title:

Initial Experiments with Duet Benchmarking: Performance Testing Interference in the Cloud

Authors:

L. Bulej, V. Horký, P. Tůma

Publication:

Proc. 27th IEEE Intl. Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)

DOI:

Year:

2019

Fulltext:

Abstract:

Accurate performance testing may require many measurements and therefore many machines to execute on. When many machines are needed, the cloud offers a tempting solution, however, measurements conducted in the cloud are generally considered unstable. In the context of comparing performance of two workloads, we propose a measurement procedure that improves accuracy by executing the workloads concurrently and using the measurements to filter outside interference. Depending on the platform used, experiments show average accuracy improvement ranging from 114% to 683% over sequential measurements on workloads running the ScalaBench suite with the Graal compiler.

BibTeX:

@inproceedings{bulej_initial_2019,
    title = {{Initial Experiments with Duet Benchmarking: Performance Testing Interference in the Cloud}},
    author = {Bulej, Lubomír and Horký, Vojtěch and Tůma, Petr},
    year = {2019},
    booktitle = {{Proc. 27th IEEE Intl. Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)}},
    doi = {10.1109/MASCOTS.2019.00035},
    pages = {249--255},
    shorttitle = {Initial Experiments with Duet Benchmarking},
}