Misleading Microbenchmarks on the Java Virtual Machines

Abstract:

Developers often use microbenchmarks to choose the most perfor- mant implementation of a method or a class. On the Java Virtual Machine (JVM), this is commonly done using the Java Microbench- mark Harness (JMH) which addresses common pitfalls of measuring code performance on the JVM. However, even using JMH guidelines cannot overcome the fundamental issue of context. Microbenchmarks inherently execute code in isolation, without interference from other application code competing for CPU re- sources, such as cache or branch-predictor capacity. On managed runtimes with tiered dynamic compilation, such as the JVM, the speculative, profile-driven nature of compilation decisions means that code performance is highly dependent on profiles collected during early execution. Because profiles usually include also branch probabilities and receiver types (besides code hotness metrics), a badly designed microbenchmark may cause the JVM to collect an unrealistic profile, resulting in aggressive, yet misleading, optimiza- tions, that would not occur in a real application. In this paper, we demonstrate how using microbenchmarks un- der conditions that induce the JVM to collect unrealistic profiles yields misleading results despite following existing guidelines. We also extend these guidelines by suggesting actions to make the microbenchmark results more representative.

BibTeX:

@inproceedings{schiavio_misleading_2026,
    title = {{Misleading Microbenchmarks on the Java Virtual Machines}},
    author = {Schiavio, Filippo and Bulej, Lubomír and Binder, Walter},
    year = {2026},
    booktitle = {{Proc. 41st ACM/SIGAPP Symposium on Applied Computing}},
    publisher = {Association for Computing Machinery},
    series = {{SAC '26}},
    location = {Thessaloniki, Greece},
    isbn = {979-8-4007-2294-3},
    pages = {9},
}

Proceedings paper