Java Industry News
Benchmark Bust-Up in Javaland
Benchmark Bust-Up in Javaland
Jan. 1, 2000 12:00 AM
(November 8, 2002) - The J2EE community got a major call to arms this week when the J2EE experts, The Middleware Company, released a report on the performance of the J2EE PetStore application running on both .NET and J2EE.
The report seemed to suggest that Microsoft's .NET performed much better than its equivalent J2EE implementation, outperforming it by a huge margin as opposed to beating it by a whisker. A time for rejoicing at the Redmond campus?
Well not quite; some Java developers believe that .NET was given a unfair edge. So what are their arguments?
The Story So Far...
Here are the circumstances, as far as JDJ News Desk has been able to establish them. The Middleware Company (TMC), the owner-managers of TheServerSide.com, approached Microsoft to aid in the production of a "fair" benchmark suite for the implementation of the J2EE PetStore application for both .NET and J2EE. The motivation for this initiative dates back to the first time Microsoft published such a report, at which time they claimed that the PetStore ran faster on .NET than it did on a leading J2EE application server.
The major J2EE companies came out in protest to against Microsoft's "findings" at that time (Oracle, IBM, and SUN for example), saying the benchmark program neglected to utilize the overall J2EE platform.
On the face of it, the re-match was straightforward: TMC would optimize and tune the PetStore application for J2EE, and Microsoft would be given the opportunity to optimize and tune their implementation for .NET. However the scales began to, shall we say, to tilt when Microsoft invited The Middleware Company to their offices, paying for all their travel and subsidiary costs to spend time at Microsoft's main offices in Seattle.
When asked by JDJ News Desk if TMC had made a similar suggestion to them to garner help configuring and optimizing their respective application servers, Sun, IBM, and BEA Systems all said that no approach was ever made to them.
"Benchmarking of this nature," comments Eric Stahl of BEA," where an application is chosen and run by someone for performance metrics, is a highly flawed methodology."
"The very reason for TPC, SPEC and the prior ECperf organization," he continues, "is to create a fair environment for running and reporting of these performance metrics in a way that properly compares the products in question. And, even with all of the overhead of a committee and its rules, the numbers can still be controversial."
"PetStore Not Designed for Performance Analysis," Says Sun
So was Sun's PetStore application the best application to use for this test?
Sun has always been quick to point to the fact that the PetStore application was never designed for performance analysis. As Glen Martin, Sun Microsystems' Lead Product Marketing Manager for J2EE, notes, "In the PetStore, given the primary goal of teaching, lucidity wins any contest. This is entirely different from the way the decision is made in real application development. People should use the PetStore to learn techniques and the application of patterns. Perhaps we haven't been clear enough on this point."
Many Java supporters have come out against this obvious oversight on TMC's part. JDJ's very own J2EE editor, Ajit Sagar, sums it up very nicely: "Comparisons are done for a purpose. If the purpose of this report was to give a fair analysis of how J2EE performs as compared to .NET, Sun as well as Microsoft should have been involved in deciding what the guidelines for the test should have been. This seems more like a "fixed" match. Was Sun approached for time, space, and resources as Microsoft was? Did they agree that Pet Store was the right application to test for performance (obviously that has not been its purpose)? Was the J2EE application deployed in Sun's facility (as the .NET one was in Redmond)? If the answer to these questions is NO, then the report is definitely one-sided and benefits Microsoft. It should be ignored by the J2EE community and treated as another marketing gimmick."
Not everyone believes, though, that TMC did a disservice to the community. Greg Leake, Lead Product Manager for .NET at Microsoft, notes: "TMC did choose a J2EE architecture that represents Sun's recommended approach using Entity Beans."
That may be very well be, but as any architect will tell you, the "recommended" approach is not always all the best approach in the real world. The Middleware Company are reputed custodians of this area, publishing well-known best practices for J2EE applications. But here lies the mystery: as Eric Stahl at BEA comments, "...the rewritten PetStore does not follow *any* of TMC's best practices. It's a joke."
TMC chose to put not one, but two leading J2EE vendors against Microsoft's .NET implementation. The two servers were not officially named, but it was obvious from the configuration files published which two servers were eventually used.
"Seriously Let Down the Java Community"
A startling fact, that both JDJ's editor-in-chief Alan Williamson and Glen Martin from Sun find hard to believe, is that "AppServer #B" failed to respond to any further requests after just 4 hours. Alan Williamson comments, "For this very reason alone, TMC had a duty to call in that particular J2EE vendor to make sure they exhausted all configuration options. This was a negligent error on a massive scale. TMC have seriously let down the Java community." Such issues wouldn't have affected the .NET implementation, as Microsoft had their own highly skilled .NET engineers poring over every single line and configuration option to ensure the best possible performance was squeezed out of it.
The question remains: why? Why would a highly respected J2EE authority be so careless? Rickard Öberg believes there is more politics to this debacle, "TMC really disqualified itself as they were recently bought by a company who has Microsoft as a strategic partner," he says - a reference to Precise Software Solutions, Inc.
"Whoever writes such a report needs to make sure that the results of it aren't released to any of the related parties before official publication," Öberg continues. He has published evidence of Microsoft having had access to the report before publication.
Leaving the potential skullduggery issues aside for the minute, from a technical level did TMC do the best possible job for the J2EE community? Rickard Öberg believes not, and has produced a rather detailed technical analysis on the PetStore implementation that the TMC used. It would appear TMC didn't even use the latest PetStore application. As Öberg points out: "The 'original PetStore' they used was 1.1.2, which is over two years old. The most recent PetStore (1.3.1) already have many of the optimizations I outline in the report."
Nigel Thomas of SpiritSoft agrees. "The comparison deals with just one possible (EJB -centric) application architecture; taking a straightforward tightly coupled/synchronous approach to application development. That's pretty much an 'entry level' approach," he continues, "that true application scalability comes from building loosely coupled applications, with non-urgent processing pushed into the background using (JMS) queues. That gives virtually unlimited scalability."
But can a benchmark be developed that would allow us to compare "apples with apples" as opposed to oranges? Many believe it can, if all parties can be brought together, and as Glen Martin says, "...a benchmark needs to be developed."
TMC has listened to the community and as a result will be re-running the tests, this time giving the J2EE application vendors the same opportunity afforded to the Microsoft team. Says Greg Leake from Microsoft, "We welcome this set of tests, and will eagerly participate with a .NET implementation of the benchmark application to showcase our technologies."
Looks like this passionate issue will go to Round #3, but this time, all parties will be participating, so the results should be far more meaningful.
JDJ News Desk looks look forward to that report.
External Resources:
The Middleware Company Report
Rickard Öberg Analysis
About Java News DeskJDJ News Desk monitors the world of Java to present IT professionals with updates on technology advances, business trends, new products and standards in the Java and i-technology space.
#51 |
David McElhaney commented on 28 Apr 2003
I'm not interested in the Sun/MS evil empire duality. "The right tool for the job" said it all. I've worked with Java for a while and it used to save my but when complexity ran up against the cross-browser compatibility constraint. I'm working with .NET now, and I find it to be very nice too. But I can't reconcile this vehement partisanship I am reading here with my own primary loyalty--to my clients.
|
#50 |
Bob commented on 18 Jan 2003
Those who lose always have an excuse
|
#49 |
Tim Nance commented on 30 Nov 2002
http://www.vnunet.com/News/1137229
The core Windows-based banking application has been rewritten in Java to make it available on any system, including over the web. One iSeries 820 server now runs the entire European operation from London, substantially reducing administration and licence costs.
"Linux runs Java much quicker than Windows. It's the natural operating system to run Java," said Evans, who added that other applications are now gradually being ported to Linux.
|
#48 |
Mark Nuttall commented on 20 Nov 2002
Why do java if one is developing on Windows? Because one might not be deploying there or doesn't want to be limited to just Windows. Look at what Fed Ex just did. BTW if performance is an issue, you can still use Java. If the person has no clue how to build good software and properly deal with performance, may the solution is to let someone else do it. They are going to have problems no matter what the software.
|
#47 |
JMA commented on 18 Nov 2002
A language that tries to be everything to all, bloats itself with APIs, and is controlled by a conglomerate of corporations with no sound strategic direction is doomed.
I like Java. Always have. But it's days are numbered.
If you are developing on Windows and performance is an issue (perhaps even time-to-market), why the heck would you even *consider* Java?
|
#46 |
Rick commented on 18 Nov 2002
This very JDJ article smacks of Microsoft FUD (fear, uncertanty, doubt). I liked the previous post pointing out that benchmarks are irrelevant, marketing hype. Focusing on customer needs and delivering solutions is what is really important. Each system has it's strengths and weaknesses, so choosing the right one for the job is the key. Pounding a square peg into a round hole takes more time, looks silly, and frustrates everyone involved. I once heard a wise co-worker once say that systems are like golf clubs, just take the time to pull the right club from the bag. It's extremely rare that any company can create the super widget that does everything for everyone. One size fits none.
|
#45 |
Ted Nanayakkara commented on 17 Nov 2002
If you need to compare J2EE and .Net, you have to rewrite petstore using only Stateless EJBs. Microsoft only has stateless components. Stateless components are the fastest. Write a stateless version in J2EE and send it to Redmond. This is all they understand.
Those J2EE developers out there, follow the J2EE blueprints and use all types of beans available in J2EE. Bill Gates will have somthing comparable to Entity Beans in about 5 years. Then we can have a one to one comparision.
In the meantime, we can laugh at the Windows developers who lack these sophisticated services we have in J2EE. Can you imagine coding a real world application without stateful, entity, MDB, etc?
|
#44 |
Reasonable commented on 16 Nov 2002
Java is slow. That's only a big shocking suprise to retards. Get over it.
"I have found that performance is not the #1 priority"
Yeah, because otherwise, Java would not be their first choice. Java has many great benefits. Why can't you admit it's shortcommings? If we can all agree that Java wasn't designed for performance, then why are you denying that it's been outperformed? Get a grip.
|
#43 |
Roland Van Liew commented on 12 Nov 2002
If this airhead Ed Roman is considered a heavy hitter in JavaLand, then Java is in serious, serious Trouble (yes, with a capital
|
#42 |
Only Rational Guy in a Group of Zealots commented on 12 Nov 2002
Whine, whine, whine. Losing to .NET is all this magazine ever talks about. I'm sick of hearing it, folks. Why don't you just admit .NET is better and close the doors on Java? You could rename your magazine to the .NET Journal or whatever.
Might as well be called that now.
|
#41 |
Brian Kiser commented on 12 Nov 2002
The light of reason shines! :) Some people get too attached to Java just because it's a good tool and is not the evil and nasty Microsoft Corporation.
Java is actually owned by the evil and nasty Sun Corporation. But at least it's not Larry Ellison. :)
|
#40 |
Rich Katz commented on 12 Nov 2002
I've read a number of accounts of the benchmark fiasco and there's something missing for me. Why don't people simply suggest what makes a benchmark fair to begin with? Or is everyone afraid to?
First, both J2EE and .Net should use THE SAME INSTANCE of THE SAME DATABASE and call the same views or stored procedures on it. There is no reason this can't be done somehow.
A) Out of the myriad of JDBC drivers for MS SQL Server surely AT LEAST ONE OF THEM will work well enough to be run on a J2EE system - especially since one (DataDirect) is recommended by Microsoft and another (Kona) is manufactured by the same people who make J2EE "Server A." There are FIVE DRIVERS for MS SQLlisted on JavaSkyline forpeetsake.
B) And if not MS SQL, run Oracle on both systems since Oracle makes an ADO provider. Either way, you have to get them to use the same instance of a database - so you aren't comparing dollars and doughnuts and you begin to eliminate network configuration problems.
It's fine to say that the benchmark isn't "database" limited. But that's no excuse and when the difference is less than an order of magnitude, the DB can make a big difference.
Second, EJB entity beans have got to go. I know it takes a long time to reprogram, but it just adds too many differences when A) there are no managed objects in the .Net version and second, and B) even if there are, we have no idea of the comparitive functionality of EBs vs. managed objects. Get rid of them - both of them. Just use stateless sessions.
Third, when do "Server C" and "Server D" and "Server E" get a chance? By that I mean C) Oracle 9i AS and D) Mort Bay with some unnamed open source EJB server - where you get to use Mort Bay's slick JDK 1.4 NIO enabled servlet engine - and E) Caucho Resin and F) Macromedia JRun Server G) Novell and H) SunONE.
You can't really say and believe that J2EE is slower than .Net unless you at least let the Hot Rods onto the track.
Fourth, let the App Server manufacturer say whether they run best on some version of Unix or some version of Windows. "Server C" might like Linux while "Server B" likes AS/400... If you pay the same amount for the Hardware, why does it matter which OS is used?
Just some thoughts.
Folks, start your engines!
Regards and the best to all
Rich Katz
|
#39 |
David Lanouette commented on 11 Nov 2002
I think most readers agree that the Java solution could have been made faster.
But, the test does demonstrate that the .NET solution is fast/scalable enough for 98% of all applications out there (on a single 8cpu machine at 1/2 the cost).
Also, I was impressed with the almost linear salability. Intel is FINALLY starting to make some fast processors, so the suitability for a given high end task will only get better.
My only serious question would be about reliability. MS has historically come up short in this measure.
|
#38 |
Tim Nance commented on 11 Nov 2002
Here is the point if you run the test in Linux Java beats .Not by 100 to 1. Oh wait .Not keeps on failing in Linux from my tests so what does this say for .Not. All .Not is just a M$ sham and it has back fired badly. Message to Microsoft you can bribe the justice department, TMC and middleware, Alan but the consumer knows the truths we had enough of VB crap and windows crashing.
|
#37 |
Philippe Leothaud commented on 10 Nov 2002
Mr Evident, if only you had taken time to read the benchmark report, you'll know that Microsoft .Net PetShop has run only twice as fast as this very very badly architected version of Java Petstore.
Furthermore, this pityful version designed (?) by TMC used very old components (EJB1.1 were released in 2000), furthermore EJBs are distributed objects, with all that this implies in terms of lack of performances, while you won't even find an object using .Net Remoting in Micrsoft PetShop!!!
All in all, I'me quite sure that some well designed Java version of petstore, like JPetstore 2.0, which will be released soon, should perform a couple of times better than PetShop on Windows, and infinitely better on any other operating system, like Unix or Linux or *Bsd, or IBMs Mainframes, which represents something like 80% of companies servers (including Microsoft, as I was told recently, which amused me a lot...)
Just one last point to end this post. Microsoft has never been a lion, but they sure are monkeys since the very beginning : they invented very few (I say thanks for SOAP), bought some stuff(remember DOS : as I told, since the beginning...) and copied much, like monkeys (OS with windows, language running in a VM, strongly typed, using a Garbage Collector --Java ? no, C#)
Monkeys, it's Evident
|