Bulletproof Web Services
Follow basic principles
Aug. 31, 2004 12:00 AM
Web services are gaining industry-wide acceptance and usage. They are moving from proof-of-concept deployments to actual usage in mission-critical enterprise applications. While Web services allow businesses to connect to partners and customers, the same flexibility and connectivity provide an increased opportunity for errors.
As companies and consumers rely more on Web services, it is increasingly important for Web services developers to know how to properly design, develop, deploy, and ultimately manage a Web services system. However, because of the inherent complexities that can arise with a Web service implementation, it can be difficult to grasp practical fundamentals and devise a step-by-step plan for Web services development.
We will look at the nuts and bolts of implementing and deploying a reliable, high-quality integration system - or rather, a bulletproof Web service. This article explains issues specific to Web services and illustrates the engineering and testing practices required to ensure complete Web service functionality. First, we will discuss the planning and design of a sample Web service. Then, we will discuss the infrastructure needed to ensure that the Web service functions properly. Whether creating Web services from scratch or integrating legacy back-end servers via Web services, the practices and principles outlined in this paper will be of great benefit.
Web Service Creation: Planning and Design
To make the discussion as concrete and pragmatic as possible, a sample Web service implementation is discussed. The example is a service for a large realtor with office branches across the country. This realtor needs to implement a Web services initiative that supports the following requirements:
- Potential and existing customers will submit contact information, desired living location, and desired price range of a home via the Web service. These users should receive a response from the server that gives them the location of the branch closest to them, as well as an estimate of the monthly mortgage. This will enable users to contact a real estate agent and begin the process of finding a home.
- Real estate agents from different branches will submit a request for a list of potential customers who are looking for homes in the local area. This will enable the real estate agents to earn business and establish contact with interested customers.
Two target requirements are needed to build the example Web service. As the name suggests, these targets are landmarks within the development process that your team should aim for. These targets will help to drive the feature set of your Web service and enable you to measure your progress. When these targets are reached, you'll know you are on the right path.
Target 1: A use case scenario shall pass after meeting these requirements:
- A dummy request is sent to the Web service for customers.
- A SOAP response is received.
- Verify that the SOAP response contains a SOAP fault.
A use case scenario shall pass after meeting these requirements:
- A valid request is sent to the Web service for agents.
- A SOAP response is received.
- Verify the SOAP response contains a list of customers that may be of length zero.
A test case is created for each target, to verify that each requirement is met. At the beginning, each test case should return an "incomplete" failure message to clearly indicate that the related feature has not yet been implemented. These test cases will continue to fail until the feature is implemented. This example starts with two targets, but this number is arbitrary. For a bigger project, you may want to have 10 targets, each one measuring an incremental step. You can track how close you are to completing your project by monitoring the test cases for these targets.
You may use various frameworks or tools to create test cases for the targets. Whatever framework or tools you decide to use, the use case scenario involves a SOAP Client sending a message, waiting for a response, and then verifying the response.
When test cases for our targets succeed, we can begin to flesh them out and verify that the new feature is implemented correctly. As seen in Figure 1, robustness requirements should be met before moving on to the next target feature.
While the target requirements drive the features set, these additional requirements ensure the robustness of the Web service:
- Normal Use: The Web service must function in the manner for which it was designed. For each operation exposed through the Web service, the request and response pair should adhere to the binding, and the XML should conform to the message description. In short, the server and client send and receive what is expected.
- Abnormal use: The Web service must function even when it is being consumed outside the lines of its intended use. An abnormal use case would involve sending a value other than those expected or not sending a value at all. For example, one application may send an XML instance document based on an older version of schema, and the receiving application may use a newer version of schema. In any case, a Web service should alert the consumer appropriately without any malfunctions.
- Malicious Use: The Web service must function even when it is deliberately and maliciously being consumed outside the lines of its intended use. For example, hackers may try to gain access to privileged information from a Web service transaction without authorization, or they may attempt to undermine the availability of the Web service. To be able to function under, or even prevent, malicious use, a Web service should have security measures in place.
- Use over time: A Web service implementation is likely to change over time. For example, perhaps a Web service exposes an application that is undergoing an iterative development process. Any Web service must continue to function properly during its entire life span, even as it is evolving.
Before jumping into the steps necessary to fulfill the target and robustness requirements above, you must be aware of the parts of the Web service that will need to be developed, tested, and verified. The initial architecture of the sample Web service will be comprised of the following (see also Figure 2):
- Application logic (or business logic): Handles requests from customers and agents, makes necessary connection to the database, and returns responses to customers and agents.
- Database: Stores relevant information about customers and agents.
- Server: SOAP-enabled HTTP server that handles serialization from XML to objects for the application logic. The Apache Axis SOAP engine is an open source SOAP implementation that can be deployed on any J2EE server.
- Proxy server: Allows for security and access management, so customers and agents have different levels of access to the available Web service.
- WSDL (Web Service Description Language) document: A description of the Web service.
- Client: The Web service client that the customers and agents will use to invoke the Web services.
Now that the basic necessities of the service to be created have been explained, you must concentrate on the foundations from which this service can be built. For Web service development to be successful, specific practices must be implemented correctly and consistently throughout your development group. This consistent application requires you to ensure that your development group has an appropriate supporting infrastructure, then ensure that the group follows a workflow from which error prevention practices are performed appropriately. Until this critical infrastructure is in place, you cannot expect a team to begin the development of a bulletproof Web service.
As seen in Figure 3, your development group must have a functioning source control system and automated build process before its members can begin writing code. A source control system and an automated build process are the fundamental requirements needed to ensure the development of quality Web services. As you will see, establishing this infrastructure provides the necessary framework for creating reliable Web services.
A source control system is a database where source code is stored. Its purpose is to provide a central place where the team members can store and access the entire source base. There are two main reasons why we require a source control system.
First, source control gives each developer the freedom and safety to write, modify, and refactor code - even when it is risky to do so on his own sandbox (the concept of a sandbox will be explained later). If a code change turns out to be undesirable, the developers can easily undo their changes by reverting back to the code in the source control. While they are working on their changes, the rest of the developers always have a working version of the code.
Second, having a source control system is a prerequisite for the nightly build process. All of the files needed for the build process should be in source control. As explained later, the nightly processes access all required files from the source control.
Most organizations do not understand how to effectively use their source control system. Many simply underuse their source control system, don't require its use at all, or have it configured incorrectly for the group environment. To put your source control system to proper use, it is important that you understand and establish guidelines for your developers. Source control systems are so important that, without a properly configured system, quality software cannot be made.
Using a Sandbox
A sandbox is an area where copies of source code and other project-related files can be stored and manipulated without affecting the master source-code base. As I mentioned earlier, the reason that each developer should have his or her own developer sandbox is so that he or she can have the freedom and safety to undertake code changes even when it is risky. In addition to these indivisual sandboxes, organizations should keep one sandbox called the build sandbox. The nightly build process will take place on the build sandbox.
None of the files in the developer sandbox should be checked out for extended periods of time. Once a developer is finished writing a feature, he or she should delete all files from his developer sandbox and then shadow a new copy. This way, the sandbox stays in synch with the source of the application.
Before checking any code into the source control system, it is vitally important that the code be compiled first. Developers should never check in code that has not been compiled. The moment code is written, developers should resolve all compilation errors and compile-level warnings.
The build sandbox should be cleanly shadowed (i.e., receive read-only copies of the master files stored in the source control system) from the source control system and deleted on a daily basis.
These principles may sound obvious, or even naive, but it is surprising to find that many software organizations lack a clear policy on what is allowed to be checked in and out of their source control systems.
Nightly Build, Deployment, and Test Process
Once a source control system is in place, the next step is to establish an automated nightly build, deployment, and test process. The main reason for establishing these processes is to be able to monitor the progress of your development. The results from the nightly build process tell you whether there are any incompatible changes in the application components. A side benefit to having a nightly build process is that, in cases where a build is shipped or released, you already have a tested build process that has been running all along. The nightly deployment process serves to set up a context in which a set of tests can be run that verify your Web service. This enables you to run the nightly test process, which, in turn, tells you whether the Web service continues to run as expected even as it continues to grow and change.
Nightly Build Process
A separate build computer (different from the computers used by developers) should be designated for the nightly build process. A separate computer is used because configurations and system settings on a developer machine can sometimes hide various dependency errors.
Each night, all of the source code and related files should be shadowed from the source control system. A scheduled task should initiate the build script that compiles the necessary components and builds the application. The results of the nightly build process must be monitored each morning. If for any reason the build process fails, the failure must be investigated and resolved so that the build process succeeds and the following processes can be performed.
Nightly Deployment Process
If the nightly build process succeeds, the nightly deployment process should be launched. This process should also be automated via a script that is initiated by a scheduled task after the nightly build process succeeds. The deployment consists of the WSDL, server, database, client, and proxy server.
First the WSDL should be created from the latest source code and exposed on a port on the same machine that the build process ran on. The WSDL should reference the most recent versions of the schemas. The application should then be exposed as a Web service on the same machine. This machine, which should be the server, should be accessible on the network and should have a reliable connection to the database. The database should be set to its default configuration during this process. The Web service client, when applicable, should be created from the latest source code and deployed. The nightly deployment process should also be monitored each night so that any errors can be detected and fixed right away.
Nightly Test Process
Finally, a nightly test process must be implemented. In this process, the newly built application, WSDL, Web service, and client are automatically tested to verify that they satisfy all the requirements and that no regressions occur in the functionality. All the test cases should be shadowed from the source control and run. Any failures must be reported and monitored the next morning. In this way, a feedback loop is established whereby errors are detected and fixed as soon as they are introduced.
In order to develop fully functional, robust Web services, you must begin with the basic principles of creating and effectively using your source control system and automatic build process. For the example used here, we took a systematic approach to creating a Web service. At each step, we defined requirements and froze them once they were met. We were able to do this through creation of verification tests that were incorporated into source control and the nightly build process. By incorporating these fundamental building blocks throughout your entire development cycle, you will be able to confidently say that you created a bulletproof Web service.