A large company found itself handicapped by an ornery snarl of siloed applications that compromised its agility, performance, and profitability. Its IT department was constantly behind schedule and over budget in hand coding point-to-point connectivity among supply chain, financials, CRM, and other packaged and custom-built legacy applications.

The solution: integrating critical business processes and applications by adopting a service-oriented architecture (SOA). Internal IT personnel and consultants engineered a loosely coupled infrastructure, with reusable services based on XML and standard Web services protocols such as SOAP and WSDL. Once the system went live, the CFO ran a routine query through his dashboard. The answer came back:

You forgot the data.

It's a playful fiction, of course, but it illustrates the perils of an SOA that focuses only on the business process interactions and application interfaces, and neglects the devilish details of data-level incompatibility among the disparate IT systems participating in those processes, including varying formats, semantics, and hierarchies.

Our hypothetical company based its SOA on a Web services-based enterprise application integration (EAI) engine. The technology worked flawlessly in enabling high-level application integration and orchestrating business processes - but it was not designed to deal with the complexities of heterogeneous, inconsistent, dirty data that lies fragmented across the enterprise.

The result: costly and time-consuming hand coding to resolve these data inconsistencies in the SOA implementation, thus violating the very promise of reusability and interoperability that is driving the movement towards SOA. The missing ingredient in this company's SOA was a data services layer built upon an enterprise data integration platform.

The SOA Opportunity
The buzz around SOA has been fast and furious. It's no wonder - organizations recognize an opportunity to slash the cost of application and middleware development and accelerate time to market by "loosely coupling" siloed applications using open standards such as Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), and Web Services Description Language (WSDL).

The widespread adoption of these standards by IT organizations and vendors alike paves the way to expose applications as component-based services for delivery over the Web. By abstracting the underlying business logic, SOA enables services to be wrapped, reused, and orchestrated to give both IT and business far greater responsiveness, flexibility, and speed of execution.

Many early SOA-based implementations have been built on EAI, and J2EE- and .NET-based middleware, including message brokers, application servers, and enterprise service buses. Increasingly however, data integration has become a primary objective. Some 76 percent of AMR Research respondents using or planning to use an SOA named process or data integration as the leading initiative, according to the August 2005 AMR Research report, "Service-Oriented Architecture: Survey Findings on Deployment and Plans for the Future." The findings reflect a growing awareness that a data integration platform can - and should - enrich an SOA with sophisticated data services beyond the scope of application integration-centric technologies.

In other words, to realize the full potential of SOA, including loose coupling and reusability, it's critical that the client application be able to access business-relevant data wherever it resides, in whatever form it is required, and in a consistent and accurate manner.

Ready for Prime Time: Service-Oriented Data Integration
Data integration technologies are ready to help SOA become a transformative force for IT. Over the past several years, data integration technology has been enhanced with built-in support for XML transformations, Web services protocols, JDBC connectivity, and Java Message Service (JMS) connectivity. Advanced data integration platforms also feature metadata capabilities driving the core of their development and run-time infrastructure. This metadata provides an abstraction of the business logic from the technical implementation, and enables them to deliver advanced data integration functionality over a data services layer to the myriad components in the SOA (see Figure 1).

For too many years, data integration initiatives, undertaken without the foundation of a data services layer, have resulted in a further proliferation of the siloed systems that they were meant to integrate. For instance, a retailer might have deployed an extraction, transformation, and loading (ETL) tool to synchronize point-of-sale data from retail outlets into an SAP financials application. A second instance of the tool might serve to move SAP financials information into a DB2 data warehouse for analysis. A third instance might work on the front end of the value chain to feed product procurement data to an operational data store.

Therefore while the retailer will have achieved data integration among targeted applications, it's still several steps removed from realizing a fluid, end-to-end data ecosystem. SOA removes these barriers of siloed development.

In a modular SOA, a data integration platform serves as another component-based service. Its functionality can be packaged and reused across multiple projects to reduce development and deployment costs. It can help an organization leverage data assets that are currently locked in mainframe, packaged, and homegrown systems through open standards. It can eliminate the need to hand-code data integration connectivity, and enables businesses to realize rapid time to value.

That's what SOA offers data integration technology. Now let's look at the flip side - what data integration does for SOA (see Figure 2).

Data Components and Services in an SOA
The most advanced SOA deployments will take advantage of both EAI and data integration technologies. SOA provides an ideal framework for these two technologies to complement one another, with EAI managing transactions and processes among applications, and the data integration platform performing the atomic-level data processing that is generally beyond the scope of EAI systems.

In fact, a common use case is where a company deploys an EAI bus and a data integration platform in an SOA to support master data management initiatives, such as customer data integration. The EAI bus drives business processes and checks customer records in the master data repository. The data integration platform creates the master data repository and populates back-end ERP systems with updated customer information transformed to the appropriate format and semantic definition.

In strategizing options and objectives for an SOA, organizations should assess and understand the functional distinctions between the two technology sets. Let's take a look at three functional components that are the exclusive province of data integration technology - universal data access, a metadata repository and services, and a data integration engine.

Universal Data Access: Scope of Data
Data integration extends the reach of the SOA and its constituent applications into virtually any data source. Prebuilt connectivity and visual mapping environments provide IT architects and developers with a mechanism to tap into information from a variety of sources, including packaged and homegrown applications such as SAP, mainframe and midrange systems such as IMS and VSAM, relational databases such as Oracle and Sybase, and unstructured and semistructured data.

Organizations can use data integration to reach into multiple systems to fetch data, cleanse and transform it into the appropriate formats and semantic definitions, and propagate it across multiple distributed systems. Its service may be invoked by, for instance, an online customer order application to trigger event-driven, read/write data updates across financials, manufacturing, and distribution in near real time.

Metadata Repository and Services: Meaning of Data
A metadata repository provides the SOA with an underlying foundation to understand the lineage of data, the ripple effects of changes, and data-related deficiencies in the architecture. The repository provides a data interaction framework to store and manage data models, transformations, workflows, and dependencies - metadata describes the data logic and its meaning. Through metadata services, data integration technology provides a means to reconcile data semantics across disparate systems; improve reporting, auditing, and data governance; and enable reuse to streamline development and accelerate deployment.

Metadata is also key in equipping organizations with an auditable record of data lineage covering all data resources, thus providing an important tool for meeting the compliance requirements of Sarbanes-Oxley and other regulations.

Data Integration Engine: Value of Data
At the core of data integration is an engine that provides organizations with a host of options for moving, integrating, and delivering data among various consumers in an SOA. Its flexibility is important for letting IT professionals architect a system optimized for "right time" data delivery, including high-volume batch data movement, near real-time capture and movement, and changed data capture - only data updated since the last service invocation.

Data integration also offers functionality to help "future-proof" an SOA against rising data volumes, and to meet the requirements for reduced data latency as well as the demands for toughened security and privacy. For example, data integration supports partitioning to optimize parallel processing on multi-CPU hardware, deployment on multi-node server grids for distributed workflow execution and fault tolerance, failover, and fortified security through authentication, authorization, and encryption.