EJB Comparison

Deployment and execution instructions

Publications

Overview

The project is a part of the performance evaluation research.

In 1999, our group launched an EJB Comparison Project for MLC Systeme GmbH, now a part of Deutsche Post under the name DP Com. The purpose of the project is to devise a sensible set of criteria for comparing Enterprise Java Beans implementations and to apply this criteria to compare four EJB servers selected by MLC (WebLogic, WebSphere, NetDynamics, GemStone/J, all of them based on EJB 1.0).

The criteria designed are reflected in a collection of roughly 60 individual tests applied to each server, not taking into account the interoperability tests. Naturally, just to run and evaluate these tests in four different environments was a non-trivial, time-consuming task requiring a lot of resources. This is actually one of the reasons why we did not dynamically move our attention towards the newly emerging EJB 1.1 servers.

The full report was 122 pages and was planned to be posted on our web site. For the few days it was online, we received a number of mostly positive reactions. The negative responses were because some of the products tested were out of date by the time the report was posted online. Those products were superseded by newer versions based on EJB 1.1. When EJB 1.1 emerged, the vendors naturally upgraded and no longer used EJB 1.0 technology.

To avoid any potentially misleading extrapolation of the results relevant to the EJB 1.0 products (plus an apology to those who might have felt their new product would do much better), we decided to only make available for a download the part of the report which outlines and describes the methodology designed for the EJB server evaluation. Strictly for illustration of the methodology, the downloadable part provides the test result numbers we have obtained for NetDynamics – this server has been chosen because, to our knowledge, it is not evolving anymore.

• The report as a PDF file (560k)
• The report as a PS.GZ file (2216k)

There are two technical issues related to the report worth to be addressed at this point:

1. The hardware platforms. Always, the hardware configuration employed in testing was more powerful than the required minimum configuration indicated in the product documentation. In fact, the tests were also run on a kind of loaded machines (e. g. Solaris CPU 2 × UltraSparc 360 MHz, 0.5 GB RAM, besides the original Windows NT: CPU Pentium 266 MHz, 256 MB RAM), yielding similar relative results.
2. Publication of the test source code. First, most of the tests address a very narrow issue explicitly described in the report so that a person familiar with the EJB standard can clearly see what is being tested. Second, the test results depend on many factors besides the source code itself (the heap size is among them) that it is in principle possible to obtain extremely different results even with the same sources. In our testing, this issue was addressed by tuning all the (known) related factors in order to achieve the best functionality (best effort test semantics). On the other hand we understand the importance of public availability of at least some of the tests source code. To illustrate our technology, we have published four selected tests, two from the compliance and two from the benchmarking area. The tests are available both in the binary and the source code form, except for the tools source code (package ejbcomp.tools), which is not relevant to the tests themselves. The source code is available as a part of the javadoc documentation (look at the "source code" statement of the documentation).

The code is compiled for the NetDynamics 5.0 Application Server and provided as a deployable jar-file plus the instructions.

• Source code
  • The source code is available as a part of the JavaDoc documentation (please look at the "source code" tags).
  • The four selected tests are:
    • Compliance
      • SesFulLifecyc – evaluates the lifecycle of stateful session beans
      • TrBmStateful – evaluates bean-managed transactions working with stateful session beans
    • Benchmark
      • RmiCall – measures the speed of method invocations with different invocation-patterns
      • NumClientsOne – measures the distribution of a number of invocations among a number of beans
• Binary
  • Jar-file for the NetDynamics Application server (md5sum: db9dbd5248a032f95ce5bfe61bd3fee0)

Deployment and execution instructions

The test suite contains four beans that can be deployed to the NetDynamics 5.0 Application Server and four clients for those beans. Each bean can be imported in NetDynamics Studio using the serialized deployment descriptor, which is enclosed in the jar-file. If a particular client is executed with no parameters, the usage information is displayed. Note that the first parameter (url of the connected server) should be always specified.

We know from the experience based on our CORBA comparison project that after the technology settles down, the frequency of product updates decreases and benchmarking results exhibit more durability. As with EJB technology, we believe this is going the happen in the EJB 2.0 context. We continue working on this exciting topic, naturally subject to our funding and manpower constraints.

We are awaiting your comments.

Publications

2000 (4)

Refereed (journals/proceedings)

Technical Reports