Web Services Integration Brokers and Enterprise Application Integration
Preparing for the future
Jul. 2, 2004 12:00 AM
Integration brokers are middleware platforms for complex enterprise application integration (EAI), enterprise information integration (EII), and business-to-business (B2B) integration.
They support flexible any-to-any integration and they provide orchestration engines that allow organizations to implement business processes that span various applications, including enterprise resource planning (ERP) and customer relationship management (CRM), and various platforms, including Windows, Linux, and Unix.
Integration brokers couple orchestration engines with flexible transformation and routing features to support multipoint application, process, service, and/or data integration. Integration broker vendors - such as BEA, IBM, Microsoft, Oracle, Savvion, SeeBeyond, Vitria, webMethods, and others - provide process-definition tools as well as connectors for interfacing to various application, operating, and networking environments.
Integration Broker Value Proposition
Integration brokers provide an environment within which business processes can be rapidly integrated and just as rapidly revised. A flowcharted orchestration can become the template for rapid business-process reengineering. An orchestration-enabled organization can rise to new challenges and reform itself rapidly into whatever new business model or value chain suits the task at hand. By redefining rules, roles, and routes within a process map, organizations can integrate, aggregate, and orchestrate existing Web services in new ways.
However, general-purpose integration-brokering infrastructures are still the exception - not the rule - in most organizations. Most orchestrated processes are limited to specific organizations, platforms, and application suites. System integrators typically use integration brokers as the basis for custom integration projects. By the same token, many enterprises invest in application suites that have embedded orchestration functionality.
Nevertheless, enterprises should begin to implement general-purpose Web services orchestration infrastructures, both within and between organizations and domains. Orchestrated process models can drive structured interactions across any and all tiers of the network application environment, including presentation, business, and integration environments.
Integration Broker Functionality
Multipoint service orchestration is the principal feature of integration brokers, and is an important ongoing requirement in most heterogeneous application environments. The term "orchestration" refers to the rule-driven flow of information, context, and control among diverse environments within distributed business processes. Integration brokers execute reusable orchestration process definitions that control multistep interactions across complex environments. When executing an orchestration, an integration broker may enforce application-level transactional context and apply rules regarding content transformation, message routing, and process invocation.
Multipoint orchestration brings isolated functionality silos together into a cohesive architecture, with integration brokers providing critical connection points in the shared communication bus. While point-to-point connections can be used to bridge isolated application silos, they don't enable efficient creation of composite business processes. In addition, point-to-point integration complicates the process of upgrading service interfaces. Coordination across diverse applications, services, and other networked resources is difficult and costly without a general-purpose integration-brokering environment.
Integration Broker Architecture
Integration brokers go by many synonymous names in today's middleware, platform, and tool market. Some of these other names include integration platforms, integration servers, orchestration servers, business process management (BPM) tools, EAI tools, enterprise service bus (ESB) middleware, process managers, process engines, and workflow environments. What all such products have in common are the following baseline components:
- Orchestration engines: All products provide runtime, server-based orchestration engines that support multipoint, any-to-any application, service, process, and/or data orchestration. All support asynchronous, messaging-based interactions among application and process endpoints, and many also support synchronous interactions. Usually, the runtime engine enables integration through message/document validation, transformation, and routing.
- Resource connectors: All provide a broad range of resource connectors for connection to diverse application, platform, middleware, transport, and data environments. Increasingly, vendors boast of offering more than 200 connectors out of the box with their integration brokers.
- Visual tools: Most vendors provide visual orchestration process definition, mapping and transformation, administration, and reporting and analysis tools. Developers often use graphical tools to specify the orchestration tasks, dependencies, and routing steps that drive runtime execution by integration brokers. Some orchestrations may involve long-running composite business processes, while others may involve short-lived atomic transactions. Often, orchestrations can be published as higher-level Web services.
Figure 1 provides a graphical overview of the components of an integration broker.
Integration Broker Features
Core orchestration features of integration brokers include:
Integration Broker Market
- Resource connection: Interfacing to applications using the Web services framework and/or various transport and middleware protocols
- Message validation: Receiving, parsing, and validating parse messages and documents originated by applications
- Policy enforcement: Applying appropriate security policies and controls - such as authentication, authorization, decryption and encryption, digital signing, and content filtering - to in-flight messages before they reach the invoked target applications
- Content mapping and transformation: Mapping and transforming the contents of messages exchanged between those platforms (e.g., translating the outputs of one application into the correct input syntax/format of another application and rewriting packet headers and contents)
- Message routing: Transmitting and routing messages to target/server applications
- Transaction tracking and coordination: Executing, controlling, tracking, and/or monitoring the end-to-end transactional workflow of messages routed between two or more application endpoints.
Fundamentally, an integration broker is any runtime rules engine that supports multipoint integration and orchestration of diverse applications, processes, services, and/or data. Even when defined this narrowly, today's integration broker market includes dozens of vendors. The integration broker market is crowded, immature, and volatile. The integration broker segment has only emerged as a well-defined niche of the middleware market in the past five to seven years, as pioneering vendors such as webMethods and Sonic Software began to provide workflow engines and tools that addressed integration requirements of e-business over Web environments.
The integration-broker market is just one component - albeit the fastest-growing segment - of the middleware market. As a technical approach, integration brokers complement such established integration approaches as object brokering (such as Common Object Request Broker Architecture [CORBA] and Distributed Component Object Model [DCOM]) and message-oriented middleware (such as Java Message Service [JMS], WebSphere MQ, BEA Tuxedo, TIBCO Rendezvous, and Microsoft Message Queuing [MSMQ]). In fact, many integration broker vendors got their start in traditional middleware approaches, and established middleware vendors such as BEA, IBM, IONA, Microsoft, Sonic, and TIBCO have decisively shifted their product focus to their integration broker offerings.
Today's core integration broker market includes two broad segments, based on how various vendors' offerings fit into broader product families: application platform vendors and middleware platform vendors.
Application Platform Vendors
Integration-broker vendors in this category provide tools that enable life-cycle development, deployment, and orchestration of services, applications, processes, and data. At the very least, these vendors provide integration brokers, application servers, and portal servers. Vendors in this segment include BEA, Fujitsu, IBM, InterSystems, Microsoft, Novell, Oracle, SAP, Sun, Sybase, and webMethods. They can be further subdivided into the major application-server vendors - BEA, IBM, Microsoft, Oracle, and Sun - and the others.
Middleware Platform Vendors
Integration-broker vendors in this category provide tools that focus primarily on orchestration of services, applications, processes, and data. These vendors provide integration brokers, resource connectors, and visual process definition and administration tools, just like the application platform vendors. However, middleware platform vendors lack application servers of their own, though they sometimes integrate with third-party application servers and integrated development environments (IDEs). Some vendors' solutions are primarily oriented to application integration (i.e., the EAI vendors), while others (the EII vendors) primarily address data integration (though many vendors' solutions do a bit of both). EAI vendors with integration brokers include Collaxa, Commerce One, Fuego, Intalio, IONA, Lombardi Software, Oak Grove Systems, Pegasystems, Savvion, SeeBeyond, Sonic Software, TIBCO, Ultimus, Vitria, and others. EII vendors with integration brokers include Ascential, Attunity, Cape Clear, Certive, Pervasive, PolarLake, TranSenda, Xaware, and others.
Increasingly, integration brokering is becoming a core feature of leading application server platforms, such as BEA WebLogic, IBM WebSphere, Microsoft Windows Server System, and Oracle Application Server. We expect this trend to accelerate as application server vendors strive to distinguish themselves through increasingly more sophisticated suites for developing, deploying, integrating, orchestrating, and managing distributed services. Web services-based middleware is fast becoming one of the core feature sets of the morphing network application environment, and every application server vendor wants to become its customers' principal "all in one" environment.
We expect to see many EAI integration broker vendors migrate toward becoming full-featured application platform vendors by adding application servers and IDEs to their core product families. In the past year, we saw webMethods - one of the pioneering integration-broker EAI vendors - take this step when it acquired The Mind Electric along with its Web services platform, middleware, and development tool offerings. We expect that many integration broker vendors will evolve in this direction through increasing integration with open source application servers such as JBoss and Apache Tomcat, and with open source IDEs such as the Eclipse platform.
Feature Innovation Raising Bar Across Integration Broker Market
We will also see continuing feature enhancements by all integration broker vendors as the industry common-denominator functionality continues to evolve. The following features will become standard in all or most commercial integration broker offerings (from both application platform and middleware platform vendors) over the next two to three years:
- Support for a broad range of EAI, EII, B2B, and human workflow requirements
- Implementation of standards-based orchestration process syntaxes, especially WS-BPEL, for enabling multivendor process definition, interchange, execution, federation, monitoring, and control
- Increasing development orientation toward visual process modeling notations - such as Unified Modeling Language (UML) and Business Process Modeling Notation (BPMN) - that abstract developers from underlying process definition syntaxes (such as WS-BPEL)
- Introduction of visual process-definition tools for use by nontechnical business process analysts
- Integration of the growing range of WSF standards (in lieu of non-WSF legacy interfaces) for identity, security, reliable messaging, transactions, semantic interoperability, and other core middleware, management, and policy functions
- Integration of Web services management (WSM) features such as lightweight message processing agents that monitor traffic performance and enforce service-level agreements (SLAs) through real-time, dynamic content routing and optimization
- On-demand downloading of software that can be used to rapidly configure a remote node to communicate with their integration brokers (a feature that will eventually be rendered unnecessary, due to universal platform implementation of the WSF stack of standards)
- Deployment of functionally specialized instances of integration brokers across enterprise application infrastructures
- Flexible configuration of specialized integration brokers throughout application infrastructures
- Inclusion of more sophisticated visual tools for defining, administering, and tracking orchestrations across multiple domains
- Interfaces to same-vendor and external UDDI and trading-partner registries for service registration, finding, and binding
- Incorporation of a broader range of process accelerator templates for various vertical markets
- Hosting of integration-brokering functionality for access by remote sites and B2B trading partners
- Integration with multivendor identity, access management, and security infrastructure through standard interfaces such as WS-Security, Security Assertion Markup Language (SAML), and others
- Administration of business rules separately from the business process orchestrations with which they are associated through incorporation of rules-inference engines that extract rules and metadata from existing application code
- Incorporation of portals and presentation-oriented orchestration development features - such as e-forms and worklists - into vendor product offerings
- Integration with e-mail, instant messaging, mobile messaging, and other asynchronous and synchronous communications environments to transmit alerts, notifications, and other messages to end users
- Access to orchestration platform functionality through same-vendor portals or, via portlets, through third-party vendor portal environments
When universal vendor support for WS-BPEL and other orchestration standards emerges, it won't make much competitive difference among integration broker vendors. Open orchestration standards will be implemented universally within platforms and tools, and will be invisible to most developers and users, just as Extensible Markup Language (XML), SOAP, WSDL, and other WSF specifications - though ubiquitous - are essentially invisible to most information technology (IT) professionals.
WS-BPEL is one of many WSF specifications that integration broker vendors must integrate natively if they wish to survive and thrive in this market. An open orchestration infrastructure depends on a broad range of WSF specifications supporting critical network application environment infrastructure features, including standards-based application messaging, service contracts, service brokering, addressing, reliable messaging, and event notification. Consequently, standards such as WSDL, UDDI, SOAP, WS-Reliable Messaging, and WS-Eventing are just as important for integration broker interoperability as WS-BPEL. These specifications will have a powerful disruptive impact on the current integration broker market, reducing vendor lock-in. Over the next three to five years, ubiquitously WSF-based EAI and B2B environments will enable any-to-any mix-and-match of integration brokers, connectors, and design/development tools from diverse vendors.
Recommendations and Conclusion
Integration brokers are central to enterprise platform, middleware, development, and management strategies. Consequently, companies must base their decisions on whether, when, and how to implement integration brokers on many business and technical considerations.
Bear in mind that all vendors discussed in this article provide the basic multipoint orchestration features discussed above: resource connection, message validation, policy enforcement, content mapping and transformation, message routing, and transaction tracking and coordination. Most vendors provide strong visual development and administration tools, diverse resource connectors, and browser-oriented business activity monitoring and control.
Beyond those core features, though, integration-broker vendors vary broadly in their development and deployment options, orchestration functionality, and legacy integration. To focus the decision process, enterprises should identify the integration scenarios in which to deploy integration brokers. Typically, integration brokers are appropriate in multipoint integration projects involving heterogeneous operating environments, application platforms, middleware protocols, and data resources.
Enterprise customers are implementing integration brokers for a growing range of middleware and orchestration requirements, and will increasingly call for standards-based multivendor interoperability. Federated, standards-based orchestration is coming to complex e-business environments everywhere. Integration brokers are the critical infrastructure that will make this possible.