Performance Evaluation
|
Introduction
Our performance evaluation research focuses on measuring and analyzing
software performance. Our activities include performance evaluation of
middleware implementations, performance analysis of modern hardware
architectures, and methods and tools for integrating performance
evaluation into software development. We intensively participate
in research projects with both academia and industry.
Resource Sharing Models
Executing software components share resources. Modeling software performance
thus means modeling resource sharing, which is commonly done for resources such
as processors or disks, but rarely done for resources such as memory caches or
collected heaps. We investigate resource sharing within the
Q-ImPrESS project.
We maintain two generic benchmarking frameworks
for investigating performance related effects due to sharing of resources
such as the processor core, memory architecture (memory caches, translation
buffers) or file system among multiple software components.
List of the measurements of the performance related effects due to sharing of resources:
Measurement Framework
Our activities involve collecting performance related observations from
heterogeneous systems. We therefore develop an extensible framework
for performance measurement that allows aggregating configurable amount of
data from configurable sources during configurable events. The framework is
part of the Q-ImPrESS project.
More details can be found in the Ph.D. Thesis of Lubomír Bulej.
You can also inspect the development version
of the framework.
Performance Prediction Validation on Random Systems
Validation of performance prediction models is typically done on a few case studies, due to a lot of manual
work needed to create the model and measure the implementation, in order to compare the predicted and actual performance.
We have proposed an approach where systems for validation are generated together with their models, allowing
validation on many systems without manual intervention.
Middleware Benchmarking
Comprehensive CORBA Benchmarks (Xampler)
Comprehensive benchmarking suite for CORBA, focus on extensibility
of the suite and breadth and depth of the evaluation, suitable for vendor
level benchmarks. See project home page
for more information, latest results and downloads. Simplified CORBA Benchmarks (Sampler)
Simplified benchmarking suite for CORBA, focus on automated execution
of the suite, suitable for user level benchmarks. See project
home page for more information, latest results, downloads,
or for uploading your own results.
Generic Results Repository (SkladR)
For storing benchmark results, we have also implemented a generic
results repository. The repository can be configured to
parse, store and plot arbitrary results. The implementation is available
for download.
Regression Benchmarking
Regression benchmarking stands as an analogy to regression testing.
By running and evaluating unit benchmarks on every version of software under
development, performance regressions can be discovered, pretty much the same
way as running and evaluating unit tests can discover functional regressions.
Acknowledgments
IONA Technologies has kindly provided a research grant for developing and running some of the benchmarks.
Archive
2007
- Linux Virtual Memory Tuning
- Performance of memory intensive applications is influenced by
virtual to physical memory mapping. We have developed an
interface through which applications with special mapping
requirements can influence the mapping strategies, implemented
as kernel modules. More details.
- Common Component Modeling Example
- We have used our tools and models in the
CoCoME
modeling contest.
2004
- OOPSLA 2004 Workshop
- We have organized the Workshop on Component and Middleware Performance
during OOPSLA on October 25, 2004 in Vancouver, British Columbia, Canada.
2003
- OOPSLA 2003 Workshop
- We have organized the Workshop on Middleware Benchmarking: Approaches, Results, Experiences
during OOPSLA on October 26, 2003 in Anaheim, California, USA.
2002
- CORBA Comparison Project, Part 3
- We have done a benchmarking project for Borland International, USA.
The final report containing the results of the measurements:
-
- IDL Tester
- A tester for CORBA IDL compilers. Generates random IDL interfaces of preset
complexity and code that links with the generated stubs. More details
on the tool page.
- IIOP Tester
- A tester for CORBA IIOP engines. Replaces a remote object reference with
a remote object proxy that intercepts IIOP messages and applies configurable
damage before forwarding the IIOP messages to the original object. More details
on the tool page.
2000
- IONA benchmarking
- In March 2000, we have done a benchmarking project for IONA Technologies, PLC, Ireland.
The project report is not publicly available.
- Automated CORBA Benchmark Suite
- In June 2000, the Automated CORBA Benchmark Suite was released.
It is now deprecated by our newer CORBA benchmark suites.
1999
- EJB Comparison Project
- Our group launched the EJB Comparison Project for MLC Systeme GmbH,
later a part of Deutsche Post under the name DP Com. The purpose of the project
is to devise a sensible set of criteria for comparing Enterprise Java Beans
implementations and to apply this criteria to compare four EJB servers selected
by MLC (WebLogic, WebSphere, NetDynamics, GemStone/J, all of them based on EJB 1.0).
- To avoid any potentially misleading extrapolation of the results relevant to the
EJB 1.0 products, we only make available for a download the part of the report which
outlines and describes the methodology designed for the EJB server evaluation and
the results for NetDynamics, which is not evolving anymore.
-
- CORBA Comparison Project, Part 2
- In June 1999, an extension of the CORBA Comparison Project was concluded.
The extension covers three more ORBs, namely omniORB 2.7.1, ORBacus 3.1.2
and Orbix 3.0, and also slightly extends the set of applied benchmarks.
- The final report containing the results of the measurements, structured as
an appendix to the original project report from August 16, 1999.
-
1998
- CORBA Comparison Project, Part 1
- In February 1998, our group launched a CORBA Comparison Project
for MLC Systeme GmbH. The purpose of the project is to devise a sensible
set of criteria for comparing ORB implementations and to apply this criteria
to compare three ORBs selected by MLC (omniORB, Orbix, VisiBroker).
- The final report containing the results of the measurements as
PS,
PS.GZ,
PDF.
Also results and
sources of the throughput test.
- The experience gained during this project lead us to sent a response
to OMG Benchmarking RFI. Our response is available as an
OMG document or
as a technical report.
Publications
2012 (1)
Accepted
|
Samuel Kounev, Philipp Reinecke, Fabian Brosig, Jeremy Bradley, Kaustubh Joshi, Babka V., Anton Stefanek, Stephen Gilmore: Providing Dependability and Resilience in the Cloud: Challenges and Opportunities,
Accepted as Book Chapter in A. Avritzer, A. van Moorsel, K. Wolter, and M. Vieira: Resilience Assessment and Evaluation, Dagstuhl Seminar 10292, Springer Verlag, 2012 |
2011 (4)
Refereed (journals/proceedings)
| PDF, Link |
Babka V., Tůma P.: Can Linear Approximation Improve Performance Prediction ?,
Proceedings of EPEW 2011, Borrowdale, UK, Springer-Verlag LNCS 6977, ISBN 978-3-642-24748-4, ISSN 0302-9743, pp. 250-264, October 2011 |
Technical Reports
| PDF |
Keznikl J., Malohlava M., Marek L., Tůma P.: Ferdinand Project Middleware List,
Tech. Report No. 2011/2, Dep. of Distributed and Dependable Systems, Charles University in Prague, January 2011 |
| PDF |
Babka V., Bulej L., Libič P., Marek L., Martinec T., Podzimek A., Tůma P.: Resource Impact Analysis,
Q-ImPrESS Project Deliverable D3.4, January 2011 |
Nonrefereed
2010 (8)
Refereed (journals/proceedings)
| PDF |
Parízek P., Yuldashev N.: Extraction of Component-Environment Interaction Model Using State Space Traversal,
In Proceedings of the 25th ACM Symposium on Applied Computing (SAC 2010), SE track, ACM, March 2010 |
| PDF |
Parízek P., Plášil F.: Assume-Guarantee Verification of Software Components in SOFA 2 Framework,
In IET Software, volume 4, issue 3, ISSN 1751-8806, June 2010 |
| PDF |
Kolb E., Šerý O., Weiss R.: Applicability of the BLAST Model Checker: An Industrial Case Study,
In Proceedings of PSI'09, LNCS 5947, ISBN 978-3-642-11485-4, pp. 218-229, January 2010 |
| Link |
Becker S., Hauck M., Trifu M., Krogmann K., Kofroň J.: Reverse Engineering Component Models for Quality Predictions,
in Proceedings of the 14th
European Conference on Software Maintenance and Reengineering,
European Projects Track, (c) IEEE, ISBN: 978-0-7695-4321-5, March 2010
(preliminary version available here)
|
| PDF, Link |
Babka V., Tůma P., Bulej L.: Validating Model-Driven Performance Predictions On Random Software Systems,
In the Proceedings of the Sixth International Conference on the Quality of Software Architectures (QoSA 2010, federate event of CompArch 2010), Springer-Verlag, LNCS 6093, ISBN 978-3-642-13820-1, ISSN 0302-9743, pp. 3-19, June 2010 |
Ph.D. Theses
| PDF |
Šerý O.: Automated Verification of Software,
Ph.D. thesis, advisor: Frantisek Plasil, September 2010 |
| PDF |
Poch T.: Towards Thread Aware Component Specifications,
Ph.D. thesis, advisor: Frantisek Plasil, September 2010 |
Technical Reports
| PDF |
Babka V., Bulej L., Ciancone A., Filieri A., Hauck M., Libič P., Marek L., Stammel J., Tůma P.: Prediction Validation,
Q-ImPrESS Project Deliverable D4.2, July 2010 |
2009 (9)
Refereed (journals/proceedings)
| PDF, Link |
Babka V., Tůma P.: Investigating Cache Parameters of x86 Family Processors,
In Proceedings of SPEC Benchmark Workshop 2009, Austin, TX, USA, Awarded with Kaivalya Dixit Award. Springer-Verlag, LNCS 5419, ISBN 978-3-540-93798-2, ISSN 0302-9743, pp. 77-96, January 2009 |
| PDF, Link |
Libič P., Tůma P., Bulej L.: Issues in Performance Modeling of Applications with Garbage Collection,
In Proceedings of QUASOSS 2009, Amsterdam, Netherlands, ACM, ISBN 978-1-60558-709-7, pp. 3-10, August 2009 |
| PDF, PDF, Link |
Babka V., Libič P., Tůma P.: Timing Penalties Associated with Cache Sharing,
In Proceedings of MASCOTS 2009, London, UK, IEEE, ISBN: 978-1-4244-4926-2, ISSN: 1526-7539, pp. 583-586, September 2009 |
| PDF |
Kofroň J., Poch T., Šerý O.: TBP: Code-Oriented Component Behavior Specification,
In Proceedings of SEW-32, IEEE, ISBN 978-0-7695-3617-0, pp. 75-83, Greece, January 2009 |
| PDF |
Kebrt M., Šerý O.: UnitCheck: Unit Testing and Model Checking Combined,
In Proceedings of the 7th International Symposium on Automated Technology for Verification and Analysis (ATVA'09), LNCS 5799, ISBN 978-3-642-04760-2, pp. 97-103, October 2009 |
| PDF, Link |
Babka V., Marek L., Tůma P.: When Misses Differ: Investigating Impact of Cache Misses on Observed Performance,
In Proceedings of ICPADS 2009, Shenzhen, China, IEEE, ISBN: 978-0-7695-3900-3, ISSN: 1521-9097, pp. 112-119, December 2009 |
| PDF |
Hocko M., Kalibera T.: Reducing Performance Non-determinism via Cache-aware Page Allocation Strategies,
Accepted at First Joint WOSP/SIPEW International Conference on Performance Engineering, September 2009 |
Technical Reports
| PDF |
Bulej L., Marek L., Tůma P.: Object Instance Profiling,
Tech. Report No. 2009/7, Dep. of SW Engineering, Charles University in Prague, November 2009 |
Master Theses
| PDF |
Marek L.: Parallel Processing and Software Performance,
Master thesis, advisor: Petr Tuma, January 2009 |
2008 (3)
Refereed (journals/proceedings)
| PDF |
Bureš T., Děcký M., Hnětynka P., Kofroň J., Parízek P., Plášil F., Poch T., Šerý O., Tůma P.: CoCoME in SOFA,
Chapter in The Common Component Modeling Example: Comparing Software Component Models, Springer-Verlag, LNCS 5153, August 2008 |
| PDF |
Bulej L., Bureš T., Coupaye T., Děcký M., Ježek P., Parízek P., Plášil F., Poch T., Rivierre N., Šerý O., Tůma P.: CoCoME in Fractal,
Chapter in The Common Component Modeling Example: Comparing Software Component Models, Springer-Verlag, LNCS 5153, August 2008 |
Technical Reports
2007 (3)
Master Theses
| PDF |
Hocko M.: Tuning Virtual Memory for Performance,
Master Thesis, advisor: Tomas Kalibera, July 2007 |
Nonrefereed
| PDF |
Bureš T., Děcký M., Hnětynka P., Kofroň J., Parízek P., Plášil F., Poch T., Šerý O., Tůma P.: CoCoME in SOFA 2.0,
Poster, Intel 12th EMEA Academic Forum, Intel Corporation, 2200 Mission College Blvd., Santa Clara, CA, USA, June 2007 |
| PDF |
Babka V., Děcký M., Tůma P.: Performance in CoCoME,
Poster, Intel 12th EMEA Academic Forum, Intel Corporation, 2200 Mission College Blvd., Santa Clara, CA, USA, June 2007 |
2006 (3)
Refereed (journals/proceedings)
| PDF, PDF, Link |
Kalibera T., Tůma P.: Precise Regression Benchmarking with Random Effects: Improving Mono Benchmark Results,
in proceedings of Third European Performance Engineering Workshop (EPEW 2006), Budapest, Hungary, Copyright (C) Springer-Verlag, Berlin, LNCS 4054, ISBN 3-540-35362-3, ISSN 0302-9743, pp. 63-77., June 2006 |
| PDF |
Kalibera T., Lehotsky J., Majda D., Repcek B., Tomcanyi M., Tomecek A., Tůma P., Urban J.: Automated Benchmarking and Analysis Tool,
in proceedings of First International Conference on Performance Evaluation Methodologies and Tools (VALUETOOLS 2006), Pisa, Italy, Copyright (C) ACM, ISBN 1-59593-504-5, October 2006 |
Technical Reports
| PDF |
Kalibera T., Lehotsky J., Majda D., Repcek B., Tomcanyi M., Tomecek A., Tůma P., Urban J.: Automated Benchmarking and Analysis Tool,
Tech. Report No. 2006/8. Dept. of SW Engineering, Charles University,Prague, June 2006 |
2005 (4)
Refereed (journals/proceedings)
| PDF, PDF |
Kalibera T., Bulej L., Tůma P.: Benchmark Precision and Random Initial State,
in Proceedings of the 2005 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2005), Cherry Hill, NJ, USA, Copyright (C) SCS, San Diego, CA, USA, pp. 853-862, ISBN 1-56555-300-4, July 2005 |
| PDF, PDF, PDF |
Kalibera T., Bulej L., Tůma P.: Automated Detection of Performance Regressions: The Mono Experience,
in proceedings of the 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2005), Atlanta, GA, USA, Copyright (C) IEEE, Piscataway, New Jersey, USA, ISBN 0-7695-2458-3, ISSN 1526-7539, pp. 183-190, September 2005 |
| PDF, Link |
Bulej L., Kalibera T., Tůma P.: Repeated Results Analysis for Middleware Regression Benchmarking,
in Performance Evaluation: An International Journal, Performance Modeling and Evaluation of High-Performance Parallel and Distributed Systems, vol. 60, pg. 345-358, Elsevier B.V., ISSN 0166-5316, May 2005 |
| PDF, Link |
Kalibera T., Bulej L., Tůma P.: Quality Assurance in Performance: Evaluating Mono Benchmark Results,
in proceedings of the Second International Workshop on Software Quality (SOQUA 2005), Erfurt, Germany, Copyright (C) Springer-Verlag, Berlin, LNCS3712, ISBN 3-540-29033-8, ISSN 0302-9743, pp. 271-288, September 2005 |
2004 (2)
Refereed (journals/proceedings)
| PDF, PDF, Link |
Kalibera T., Bulej L., Tůma P.: Generic Environment for Full Automation of Benchmarking,
in proceedings of Net.ObjectDays 2004, First International Workshop on Software Quality (SOQUA 2004), Erfurt, Germany, Copyright (C) 2004 tranSIT GmbH Ilmenau, Germany, pp. 35-41, ISBN 3-9808628-3-6 ; also in Testing of Component-Based Systems and Software Quality, Proceedings of SOQUA 2004 and TECOS 2004, LNI 58, GI 2004, pp. 125-132, ISBN 3-88579-387-3, ISSN 1617-5468, September 2004 |
| PDF, PDF, Link |
Bulej L., Kalibera T., Tůma P.: Regression Benchmarking with Simple Middleware Benchmarks,
in proceedings of the 2004 International Performance Computing and Communications Conference (IPCCC 2004), International Workshop on Middleware Performance, Phoenix, AZ, USA, Copyright (C) 2004 IEEE, Piscataway, New Jersey, USA, pp. 771-776, ISBN 0-7803-8396-6, ISSN 1097-2641, April 2004 |
2003 (1)
Refereed (journals/proceedings)
| PDF, PDF |
Buble A., Bulej L., Tůma P.: CORBA Benchmarking: A Course With Hidden Obstacles,
In the Proceedings of the 2003 Internation Parallel & Distributed Processing Symposium (IPDPS 2003), Workshop on Performance Modeling, Evaluation and Optimization of Parallel and Distributed Systems (PMEOPDS 2003), Nice, France, Copyright (C) 2003 IEEE, Piscataway, New Jersey, USA, ISBN 0-7695-1926-1, ISSN 1530-2075, pp. 279, CDROM DATA/W18_PMEO_11.PDF, April 2003 |
2002 (3)
Refereed (journals/proceedings)
| PDF |
Tůma P., Buble A.: Overview of the CORBA Performance,
Proceedings of the 2002 EurOpen.CZ Conference, Znojmo, Czech Republic, September 2002 |
Master Theses
| PDF, Link |
Strnad P.: IDL Source Generator and IDL Compiler Testing Tool,
Master Thesis, advisor: Petr Tuma, April 2002 |
| PDF, Link |
Kuhn J.: IIOP Engine For Testing CORBA Implementation Robustness,
Master Thesis, advisor: Petr Tuma, September 2002 |
2001 (2)
Refereed (journals/proceedings)
| PDF |
Tůma P., Buble A.: Open CORBA Benchmarking,
Proceedings of the 2001 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2001), Orlando, FL, Copyright (C) SCS, San Diego, CA, USA, July 2001 |
Technical Reports
| PDF |
Tůma P., Buble A.: Technical Report on Open CORBA Benchmarking,
Tech. Report No. 2001/1, Dep. of SW Engineering, Charles University, Prague (slightly modified form appears in the SPECTS '01 conference proceedings), March 2001 |
2000 (4)
Refereed (journals/proceedings)
| PDF |
Procházka M.: Advanced Transactions in Enterprise JavaBeans,
Proceedings of EDO 2000, Davis (CA), USA, Springer Verlag, LNCS, October 2000 |
Technical Reports
| PDF |
Distributed Systems Research Group: EJB Comparison Project,
Final Project Report, Public Distribution Version, February 2000 |
| PDF |
Procházka M.: Extending Transactions in Enterprise JavaBeans,
Tech. Report No. 2000/3, Dep. of SW Engineering, Charles University, Prague, 2000 |
| PDF |
Procházka M., Tůma P., Pospíšil R.: Enterprise JavaBeans Benchmarking,
Tech. Report No. 2000/4, Dep. of SW Engineering, Charles University, Prague, 2000 |
1999 (2)
Technical Reports
| PDF |
Distributed Systems Research Group: CORBA Comparison Project,
Project Extension Final Report, August 1999 |
Master Theses
| PDF, PDF |
Buble A.: Comparing CORBA Implementations,
Master Thesis, advisor: Petr Tuma, August 1999 |
1998 (2)
Technical Reports
| PDF |
Distributed Systems Research Group: CORBA Comparison Project,
Final Project Report, June 1998 |
| PDF |
Plášil F., Tůma P., Buble A.: CORBA Benchmarking,
Tech. Report No. 98/7, Dep. of SW Engineering, Charles University, Prague (in a slightly modified form submitted to OMG as the Charles University Response to ORBOS Benchmark RFI as the OMG document bench/98-10-04), October 1998 |
|
Home
People
Research
Projects
Publications
Teaching
Links
Contact
