Working with service-oriented applications is a lot like doing your taxes because: there's really no avoiding it, you need to keep track of a number of different factors, you must maintain a record of data to protect against audits and plan for the future, and you'd better catch any mistakes or you'll have a problem on your hands faster than you can spell "SOA."

Unless you implement management and monitoring capabilities for your loosely coupled system, you expose yourself to any number of potential issues that can cripple a production SOA application. Worse yet, you might unleash problems that you won't even know about until they've impacted your business. Sound daunting? It can be. The good news is that there are solutions to solve the taxing problem of SOA management.

A great example of how to implement SOA management the right way can be found at H&R Block Financial Advisors, the Detroit-based subsidiary of H&R Block, Inc.

A Leading Financial Services Company
H&R Block Financial Advisors offers investment planning, advice, and related financial services and products. With more than $28 billion in assets under management, they are one of North America's largest broker/dealers.

H&R Block Financial Advisors receives a wealth of leads from the parent company's tax services organization. Consider that H&R Block handled more than 21,000,000 tax clients in 2003. Many, but not all, of these tax clients are excellent prospects for the financial advisors. To ensure that only qualified leads are routed expeditiously to the right financial advisor at one of more than 250 offices across the country, the company requires sophisticated lead-routing capabilities. When a hefty license renewal fee came up for their previous lead management system, they took the opportunity to make the move to SOA and implemented the services-oriented Client Acquisition System, or CAS. CAS aggregates roughly 3,000,000 leads in a single tax season. It qualifies them and then routes them based on criteria specified by the branch managers at each financial advisor office.

Here's how it works. CAS executes a series of orchestrated Web services to pull together the pieces of information required to qualify, route, and assign leads to a financial advisor. TIBCO BusinessWorks is the CAS orchestrator that manages the execution of 15 .NET Web service requests per lead. If a lead is qualified, a final Web service call is made to the CRM system in order to process the lead and assign it to an advisor. The financial advisors have specialized views in their CRM application that enable them to see new leads and to filter prospects based on the criteria of their choosing.

Handling Exceptions
As with all production systems, CAS is not immune to exceptions. The number and variety of data sources, combined with the evolutionary nature of distributed applications, causes errors and unexpected conditions that mandate comprehensive exception-handling capabilities.

"Exception management has always been and will always be a problematic area for IT," said Scott Thompson, head of application development for H&R Block Financial Advisors. "Unfortunately, even though it's an absolute necessity for most systems, it tends to pull our attention away from the things the business really cares about, like routing leads. By implementing a dedicated SOA management system we not only had more time to focus on the leads, but also established a framework we can use across the enterprise."

H&R Block uses AmberPoint SOA management software to manage exceptions. AmberPoint's agent-based architecture provides centralized visibility into system-wide, services-based interactions. It identifies unique message flows from end to end and scans messages for user-defined exception conditions, including those that span multiple service interactions. Upon detecting an exception, it alerts individuals and other systems. At the same time, it automatically assembles queryable message trails for inspection by users. Because all messages are available in a single facility, the company's developers don't have to search multiple log files to identify the sources of problems. All of the captured operational data is provided via standard SQL reports.

To ensure that H&R Block Financial Advisors's lead-routing system operates smoothly, the management system monitors the traffic, checks fault codes, and handles exceptions as required. Some exceptions need only to be logged, while others require immediate resolution. For example, if the source system sends invalid data or CAS can't execute a qualification for determining the quality of a lead, AmberPoint alerts the developer assigned to support the source.

The management system significantly reduces the time it takes H&R Block Financial Advisors to detect, diagnose, and resolve system errors.

"Web services complicate the management of exceptions because of their 'black box' nature," explained Thompson. "Prior to AmberPoint, we relied very heavily on our consuming applications to tell us when errors occurred. In particular, HTTP-related errors were very difficult for us to detect and place metrics around. It was also hard to justify taking the time required to develop our own custom solutions to provide the level of visibility we knew we needed. AmberPoint provides that visibility, usually identifying errors before our consumers know they occurred. We can also look back over time and identify problematic areas that may need to be addressed in future releases of our services.

"Our management system gives me visibility into the traffic so I can see the actual information that's passed into the service and the information that came back out," said Thompson. "We also benefit from the system's ability to serve as a QA mechanism. If there's a bug in the code, we can fix it, redeploy the code, and use AmberPoint to resubmit the same message to see if the problem persists."

Since taking CAS into production, H&R Block has caught and handled more than a million exceptions - including errors and other system events. For example, an ignored lead is captured as an exception and quickly remedied.

Ensuring the Operational Health
Service level management is another pressing concern for SOA. By managing service level agreements, or SLAs - the formal guidelines they set for acceptable performance levels of applications and services - H&R Block is able to identify and mitigate performance bottlenecks before they can impact the business. They also use this detailed service level data to fine-tune the system for better performance.

Thanks to this greater visibility and the ability to catch performance issues before service level agreements are violated, CAS has achieved 100 percent availability. It has not missed a lead since going into production and has performed flawlessly through the peak loads of this year's tax season.

Abstracting the Management Layer
Whether you build your own SOA management system or purchase one, you'll want to give careful thought to the underlying mechanisms used to provide management capabilities. If you hard-code the management logic into the Web services themselves you'll be challenged to maintain your management capabilities as your system evolves. Having your Web services and clients embed proprietary headers into SOAP requests and responses would limit your management capabilities to those system components over which you have direct control. So, for example, you'd have no visibility into the performance of a partner's Web services. The best approach, therefore, is to abstract the management layer by using Web service intermediaries. With this approach, all of the management policies reside in the intermediaries, thus enabling your system to grow quickly and flexibly - and without added development-time constraints.

What's more, by abstracting the management layer you enable your developers to focus on the business problem at hand.

A Healthy Return
After its first full tax season in production, CAS has already delivered an impressive return on H&R Block's investment in SOA. As compared to the previous lead-routing system, CAS has reduced their support staff from ten dedicated resources to three shared ones. License costs are a fraction of what they used to be. Development costs are significantly reduced as well - as much as 20 percent lower. Leads are routed quickly and effectively to the advisor best suited to work with the prospective customer. In 2005, the volume of leads rose to 3,000,000, 30 percent of which were routed to financial advisors. With the old system it could take as long as 24 hours for a lead to be delivered from a tax office to a financial advisor. Even under the highest load, it now takes only seconds. Most important of all, the company's revenues have risen significantly, due in large part to the better-qualified leads they're sending to their financial advisors.