Middleware Benchmarking: Approaches, Results, Experiences OOPSLA 2003 Workshop October 26, 2003 Anaheim, California, USA
The workshop summary article is in preprint phase

The goal of the workshop is to help advance the current practice of gathering performance characteristics of middleware implementations through benchmarking. The workshop will serve as a meeting point between middleware developers and middleware users as two representative groups that are typically involved in middleware benchmarking but tend to have different requirements. Positions are solicited especially from people with previous or impending benchmarking experience.

The participants of the workshop will identify requirements and obstacles of middleware benchmarking and form a position on issues such as designing a framework that would allow the design, running and evaluating a diverse range of benchmarks over a diverse range of middleware, designing benchmark criteria that would allow for a meaningful comparison of results collected over different platforms, designing a framework suitable for regular regression testing, or providing means to verify the benchmarking results and their applicability to specific usage situations.

Identified Questions

• What do you think is the biggest plus or minus of an existing benchmark or benchmarking framework ?
• What would you want a benchmark standardization effort to focus on ?
• What is the impact of various object oriented designs in a benchmark ?
• Is there anything besides performance that a benchmark should measure ?
• How should requirements on middleware performance be characterized ?
• How should workload and background load be characterized ?
• How should a platform independent benchmark be specified ?
• What is needed to use a benchmarking framework for regular regression testing ?
• What are the guidelines for carrying out performance and scalability evaluation studies ?
• How to do a meaningful comparison of results collected over different platforms ?
• How to verify the benchmarking results and their applicability to specific usage situations ?

Schedule

To reflect the focus of the submissions, the schedule divides the workshop into two parts. The morning part groups presentations related to middleware benchmark construction, whereas the afternoon part groups presentations related to workload generation and characterization. The schedule was arranged to provide time for debate after each presentation.

The schedule ends with a preparation of the workshop statement, which will become a basis for the workshop poster and later also for the workshop publication. To help achieve convergence, a set of open questions relevant to middleware benchmarking will be raised by the participants during the opening of the workshop, and each presentation will be asked to conclude with explaining which questions and how it addresses.

Submissions

The organizing committee will disseminate all submissions to the workshop attendees prior to the workshop and select position papers and experience reports to be presented at the beginning of the workshop. The submissions will also serve as a base for a joint workshop report, to be published in the Concurrency and Computation: Practice and Experience journal by Wiley.

The statements of interest, position papers and experience reports should be mailed to the organizing committee at oopsla-workshop@d3s.mff.cuni.cz. Any of the DOC, PDF, PS or RTF formats are accepted, for formatting instructions please see the ACM instructions.

Important dates

Organizing committee

• Paul Brebner, CSIRO, Australia
• Emmanuel Cecchet, INRIA Rhone-Alpes, France
• Julie Marguerite, ObjectWeb Consortium, France
• Petr Tuma, Charles University, Czech Republic (chair)