Overview

Software performance tends to be easier to observe after the fact than to design up front. Focusing on the implementation phases of software development, we work on methods that help improve developer awareness of software performance.

Performance Unit Testing strives to construct specialized unit tests that validate performance. The obstacles we address include portable test construction and robust evaluation of noisy test results.

Performance Documentation incorporates information on code performance into standard code documentation. Performance is collected from corresponding performance unit tests and displayed interactively in standard documentation.

Performance Unit Testing

Compared to functional unit testing, performance unit testing is more difficult, partially because correctness criteria are more difficult to express for performance than for functionality. Using the Stochastic Performance Logic (SPL) [1], we aim to express assertions on code performance in relative, hardware-independent terms. Using the performance unit testing tools, these assertions can be automatically validated [3] [5]. Besides performance unit testing, we experiment with using the performance unit tests to generate data for software documentation extended with performance information [4]. Other research includes incorporating performance awareness into adaptive applications [2].

Example

Stochastic Performance Logic (SPL) allows the developer to capture assumptions about performance of a code fragment. For example, to express that a proprietary sorting method is faster than the one provided by the Java Class Library, we can execute a test with the following annotation (somewhat simplified syntax):

@SPL("for n in 100, 200, 500, 1000 SELF(n) < java.util.Arrays#sort(n)")
public static void proprietarySort(int[] data) {
	...
}

As another example, it is possible to also express than newer version of the method is faster, helping to prevent performance regressions:

@SPL("for n in 100, 200, 500, 1000 SELF@rev951(n) < SELF@rev840(n)")
public static void proprietarySort(int[] data) {
	...
}

Complete examples are included with the tools.

Tools

The core tool is a command line utility to execute the performance unit tests. The utility scans a given project and locates and executes the tests specified using annotations. The results are provided in the form of an HTML report with a quick overview and a detailed information page for each test, including visualization of the measurements. The utility can cooperate with both Git and Subversion to fetch particular project versions used as baselines, it can also use SSH to execute the measurements on a remote host, which can help improve measurement stability.

In addition to the core command line utility, we also provide plugins for the Eclipse integrated development environment and the Hudson continuous integration server. The Eclipse plugin provides project configuration, annotation editor with content assist, and interactive results browser also for results from Hudson.

Tool Link Description
Documentation User Guide, Developer Manual
Core tools Packaged Releases, Source Repository Core command line utility for executing performance unit tests.
Eclipse plugin Installation Guide, Update Site, Source Repository Eclipse plugin for syntax highlighting and expression completion, unit test execution, report browsing.
Hudson plugin Packaged Releases, Source Repository Hudson plugin for seamless performance unit test support in continuous integration.

We also make available research prototypes under development, listed below.

Tool Link Description
Experimental SPL Implementation Source Repository Prototype implementation of the core tools for performance testing, with support for runtime data collection based on DiSL.
Runtime Adaptation Prototype Source Repository An agent for runtime collection of performance data that can be used to control self-aware applications that adapt to their own performance [2].
Formula Evaluation Engine Source Repository A standalone module for evaluating SPL formulas that supports multiple interpretations. The module contains no probes, instead it operates on externally provided data.
SPL Integration with JMH Source Repository A Maven plugin to ease integration of the JMH microbenchmarking harness with SPL formulaes.

Experiments

We have performed a retroactive study where we have picked an existing software project, augmented it with tests and evaluated the results. Our project of choice was JDOM, a software package for accessing, manipulating, and outputting XML data from Java code. With about 15000 lines of code, it is of reasonable size for an experiment that requires frequent compilation and manual code inspection. With over 1500 commits spread across 13 years of development, it provided ample opportunity to observe performance regressions. JDOM is an open-source project with public source code repository. Details of the case study can be found in [3].

The table below contains the results from the measurements on three different machines.

Machine OS & Java VM version Test reports1 Raw data2
Intel Xeon (2.33GHz, 8GB) Fedora 18 64bit, OpenJDK 1.7 Browse on-line, ZIP archive ZIP archive
Intel Pentium 4 (2.2GHz, 512MB) Fedora 18 32bit, OpenJDK 1.7 Browse on-line, ZIP archive ZIP archive
Intel Atom (1.6GHz, 1GB) Windows XP SP2 32bit, Oracle HotSpot 1.7 Browse on-line, ZIP archive ZIP archive

1 Open index.html in the ZIP archive to see the overview page of the HTML report.

2 The data files are plain text, one execution time per line. The first line of each file contains the measurement identification, including machine, method, repository, revision, workload generator and date. To process the data files in R, use read.table("filename.dat")[,1], which gives access to the measured values as a vector of execution times in nanoseconds.

Performance Documentation

To increase performance awareness in software development, we use the performance unit tests to enrich reference documentation with performance related information. To this end, our tools extend the standard Javadoc generator to include an interactive performance browser with each method that has a performance unit test. The tools execute measurements on the fly, gradually increasing the accuracy of presented results [4] [6].

Tools

Tool Link Description
Performance Documentation Source Repository An extension of the standard Javadoc generator with performance related information.

Acknowledgments

This work has been supported by the EU project 257414 ASCENS, the GACR project P202/10/J042 FERDINAND, and by the Charles University institutional funding SVV-2012-265312 and SVV-2013-267312.

Contact

Petr Tůma petr.tuma<at-sign>d3s.mff.cuni.cz
Vojtěch Horký vojtech.horky<at-sign>d3s.mff.cuni.cz

References

  • [1]Bulej L., Bureš T., Keznikl J., Koubková A., Podzimek A., Tůma P., Capturing Performance Assumptions using Stochastic Performance Logic, In proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering (ICPE), Boston, MA, USA, ACM, ISBN: 978-1-4503-1202-8, pp. 311-322, April 2012. PDF, Link
  • [2]Bulej L., Bureš T., Horký V., Keznikl J., Tůma P., Performance Awareness in Component Systems: Vision Paper, In proceedings of the 26th IEEE Computer Software and Applications Conference Workshops (COMPSACW), the 4th IEEE International Workshop on Component-based Design of Resource Constrained Systems (CORCS), Izmir, Turkey, IEEE, ISBN: 978-1-4673-2714-5, pp. 514-519, July 2012. Link
  • [3]Horký V., Haas F., Kotrč J., Lacina M., Tůma P., Performance Regression Unit Testing: A Case Study, In proceedings of the 10th European Performance Engineering Workshop (EPEW 2013), Venice, Italy, Springer, LNCS 8168, ISBN: 978-3-642-40724-6, ISSN: 0302-9743, pp. 149-163, September 2013. Link
  • [4]Horký V., Libič P., Marek L., Steinhauser A., Tůma P., Utilizing Performance Unit Tests To Increase Performance Awareness, In proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE), Austin, Texas, USA, ACM, ISBN: 978-1-4503-3248-4, pp. 289-300, January 2015. Link
  • [5]Trojánek T., Capturing Performance Assumptions using Stochastic Performance Logic, Master Thesis, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic, 2012. Link
  • [6]Náplava J., PerfJavaDoc: Extending API Documentation With Performance Information, Bachelor Thesis, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic, 2015. Link
Logo of Faculty of Mathematics and Physics
  • Phone: +420 951 554 267, +420 951 554 236
  • Email: info<at-sign>d3s.mff.cuni.cz
  •  
  • How to find us?
Modified on 2016-12-06