SYS-CON MEDIA Authors: Yeshim Deniz, Kevin Benedict, Gilad Parann-Nissany

Related Topics: SOA & WOA

SOA & WOA: Article

WebSphere vs WebLogic: IBM and BEA Spar Over SPEC Results

SPECjAppServer2004 Results Published, But What Do They Mean?

Benchmarks can mean whatever you want them to mean, it has always seemed. Although useful as a rough guide to performance, and sometimes price/performance, technology companies are famous for interpreting complex benchmark results as victories over their competition and them employing visual aids such as planes, snails, and automobiles to demonstrate their point.

The latest benchmark brouhaha may be over the newly developed SPECjAppServer2004 benchmark from SPEC (the Warrenton, VA-based Standard Performance Evaluation Corp.) This benchmark  is the latest in the "Java Client/Server" category, and replaces SPECjAppServer 2002, which has now been retired. It "is designed to measure the performance of J2EE 1.3 application servers, (and) includes an enhanced workload by adding a web tier, JMS, and other changes to SPECjAppServer2002," according to SPEC.

And it has lent itself to spectacular success by IBM, according to a recent release coming out of Somers, NY. In fact, IBM's WebSphere Application Server "outperformed the competition by 64 percent," according to the release. Competitor BEA was quick to dispute this claim, however, pointing to its own success with its WebLogic on this benchmark.

IBM executive Sandy Carter (pictured) said “these latest results further demonstrate the industry leading capability of IBM WebSphere Application Server,” said Sandy Carter, Vice President, IBM WebSphere Software. And the IBM release went onto say that the "results submission involved more than 22,000 concurrent clients and produced more than 2,921 complex business transactions per second. The IBM submission represented a complete IBM solution bringing together the latest version of WebSphere Application Server software, DB2 Universal Database and IBM System p5 550 servers running SUSE Linux. This powerful solution outperformed a combination of BEA and Oracle software running on Sun Solaris and at significantly lower total cost of ownership across the full configuration."

BEA executive Eric Stahl (pictured), the company's Senior Director of Investor Relations, begs to differ. He referred to IBM's results as "inefficient," and has developed charts that show WebLogic as the clear winner in this contest. In his blog, Stahl says "the new (SPECjAppServer2004) result from BEA sets a new scalability world record, beating the previous record which was already held by BEA," and he provides a chart in his blog to illustrate his point.

Neither Oracle nor Sun had a comment about the IBM release by the time this story was being posted. Oracle has also not published its results on this benchmark, although Sun has. The results table is available to anyone at the SPEC website.

Which leads to the price/performance question. Unlike the 2002 benchmark, the new one does not include a price/performance metric, so ladies and gentlement, please re-start your engines. IBM says it based its 64% margin of victory on the following comparison:

More Stories By Roger Strukhoff

Roger Strukhoff (@IoT2040) is Executive Director of the Tau Institute for Global ICT Research, with offices in Illinois and Manila. He is Conference Chair of @CloudExpo & @ThingsExpo, and Editor of SYS-CON Media's CloudComputing BigData & IoT Journals. He holds a BA from Knox College & conducted MBA studies at CSU-East Bay.

Comments (2) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Mirror World 11/15/05 05:16:45 PM EST

Trackback Added: Performance at a cost; Sys-con are trying to fan the flames of the current App Server performance wars - probably in an effort to drum up some interest in Friday's App Server shootout - at which I'll be representing Sun.

It's encouraging to see that both IBM and BEA are n

SOA Web Services News Desk 11/12/05 11:25:45 PM EST

IBM reported a 64% margin of victory over the competition on a new benchmark developed by Standard Performance Evaluation Corp., or SPEC. But when apprised of this claim, BEA executives said 'whoa, just a minute, podner!' So what do these results really mean?