I just got back from the Gartner Application Integration show. This is perhaps the seventh of these conferences I attended, including the first one back in 1998. The good news is that the conference was packed and application integration seems to be making a strong resurgence with a little help from something called Web services. The bad news is that many of the attendees appeared confused, and the world of application integration and Web services/SOA seems more complex and difficult to grasp for the rank-and-file IT person than it was in the past. There don't seem to be as many clear answers these days, which is too bad because the problems seem to be getting bigger and integration is becoming a top-three IT priority according to the analysts.

The real surprise for me was the backlash against the concept of ESB (enterprise services bus) and its proper application within the enterprise as well as in the world of SOA. It seems to be falling off the current buzzword list as enterprises press forward with their SOA planning and implementation work, choosing more Web services?compliant messaging infrastructures instead.

In fact, I was most surprised to hear an analyst admit during a presentation that there was a flaw in the ESB concept, as end user organizations may over-integrate proprietary ESBs. This comment points to the fact that many ESBs will layer on top of existing messaging systems already in place, and may also be redundant with newer, more Web services?compliant approaches. In other words, you could end up with your traditional messaging system alongside your new ESB messaging system, as well as a new SOA infrastructure, all with redundant messaging capabilities.

The analysts could have a point there when considering the long-term strategic directions of SOAs. Indeed, many are calling ESBs "transitory" technology, something that will bridge enterprises to Web services, but perhaps not be the end solution.

Of course the ESB guys would argue that their ESB solution is indeed Web services/standards-based, albeit the underlying technology is based in Java or proprietary messaging systems. If you deal with the technology through standard interfaces, you should not be concerned with what happens behind the scenes. Moreover, they have all vowed to keep up with emerging Web services standards.

What's driving the debate even further is the appearance of new SOA infrastructure players with ESB-like capabilities, such as Blue Titan and their Network Director product with embedded queue technology in its Web services routers. Like the ESBs, they provide guaranteed message delivery, but they are also compliant with the current version of WS-ReliableMessaging.

The Essence of ESBs
I've heard the term ESB from both the analysts and vendors (both claiming ownership, by the way) to describe a technology stack that places service-oriented interfaces on top of messaging systems, such as JMS. Instead of invoking the Java interface to push and pull messages from a queue, you leverage a Web services interface. Going further, many of the ESB vendors have added "traditional EAI" features to their stacks, including transformation, routing, flow control, and process integration. Indeed, many have described ESB as kind of an EAI light.

Thus, those who employ an ESB would use it primarily for information-oriented integration, leveraging the ESB to move information between applications, which is traditionally the role of a messaging system. They also add transformation, routing, and other information brokering capabilities. However, most ESBs lack service- or behavior-based integration capabilities and the ability to effectively create and manage composite applications through this infrastructure. They do support simple orchestration, but again, typically around information movement.

The lack of true service-based integration is really the kicker for me when considering an ESB for use in an SOA. I believe the real value of creating an SOA and moving to Web services is the ability to reuse services inside and outside of an enterprise and create new applications by assembling services. This is how SOAs make you money. Also, SOAs provide us with the ability to create orchestration layers, abstracting services, and information flows for the purposes of creating business processes.

Thus, the purists are putting forth the argument that most SOAs should leverage pure Web services standards, otherwise integration and interoperability will be compromised. They further state that enterprises are going to double or triple up on messaging layers, implementing a JMS-based messaging system as well as a pure Web services?compliant messaging system at the same time. Moreover, ESBs typically exist with legacy messaging systems that are already in place. Furthermore, now that WS-ReliableMessaging is almost ready for primetime, as well as WS-Reliability (Sun's version, now ratified by OASIS), we're nearing the time when we can build Web services?compliant messaging systems without the need for an ESB. Thus SOA architects have other options today.

So, What's the Deal?
Clearly, building an SOA is one of the most confusing things to do these days. Not because the technology is so difficult to leverage; it's not, but there are many decisions to make, such as wading through competing technology as well as overlapping and numerous standards.

ESBs, at their essence, are reinvented messaging systems, but do have value for those who need messaging systems today and want tighter integration with their new service-based infrastructure. They do raise questions, however, as to their long-term value in light of newer, more Web services?compliant technology.

ESB vendors, however, are not going to stand around and watch their technology become irrelevant. Count on them to morph their product to meet changing expectations, including addressing emerging and existing Web services standards. Indeed, almost all ESB vendors have aligned with Web services standards, including WS-ReliableMessaging or WS-Reliability.

The key issue is that of fit. If you already have an existing enterprise middleware layer and are moving to service-oriented or Web services?compliant integration, leveraging an ESB is a step you may be able to skip. However, ESBs do seem to be a good fit for those enterprises performing simple information movement between stovepipes that don't need the heavy duty nature of more traditional integration technology. Once again, you have to map the appropriate technology to the problem, as well as consider longer-term strategic direction.