What Do R Benchmarks Measure: Toward a Performance-Characterized Benchmark Suite for R

Abstract:

Many programming language implementations have a benchmark suite that consists of a fixed set of programs with a harness and clear maintenance. R has none. Instead, R implementations are evaluated on ad hoc sets of programs and harnesses with different timing conventions. This paper is a step toward a benchmarking suite for R. We collect 118 R programs from 8 repositories and put them under a single harness. Next, we characterize their performance profiles, which show the share of the run spent in different regions of the R virtual machine (e.g., bytecode interpreter, memory management, builtin functions). This allows us to classify benchmarks by what they exercise so we can create suites that target a specific use such as compiler optimizations. On three R compilers, the speedups follow the profiles. They concentrate in the interpreter-bound programs and vanish in the ones dominated by native code. The profiles also cut the cost of running a suite. Dropping the 53 programs dominated by native code or builtins leaves 65 programs and saves 51.7% of the running time of the corpus.

BibTeX:

@inproceedings{kocourek_what_2026,
    title = {{What Do R Benchmarks Measure: Toward a Performance-Characterized Benchmark Suite for R}},
    author = {Kocourek, Matěj and Říha, Filip and Tušla, Oliver and Donat-Bouillud, Pierre and Křikava, Filip and Vitek, Jan},
    year = {2026},
    booktitle = {{Proceedings of the 18th ACM SIGPLAN International Workshop on Virtual Machines and Intermediate Languages}},
    publisher = {Association for Computing Machinery},
    series = {{VMIL '26}},
    location = {New York, NY, USA},
    doi = {10.1145/3840562.3844972},
    isbn = {979-8-4007-2932-4},
    pages = {39--49},
    url = {https://dl.acm.org/doi/10.1145/3840562.3844972},
    shorttitle = {What Do R Benchmarks Measure},
}

Proceedings paper