WSJ Management

Web services provide organizations with a flexible, standards-based mechanism for deploying business logic and functionality to distributed consumers. Consumers, whether internal or external, can access necessary components such as account information, credit card validation, and much more. When business functionality is distributed, however, quality management becomes imperative. Mission-critical functions, sensitive data transmission, and fee-based services must work quickly and accurately for all users all the time. To ensure that this high level of quality is met, organizations must employ strong test processes to ensure that necessary Web services are developed and deployed to meet the highest standards.

Defining the Service
Quality starts with requirements. In order to design and deliver Web services, developers must know what functionality is needed and under what conditions. The more specific the requirements, the better the development effort. "The Web service must be fast" is useless as a requirement as it is subject to interpretation. "The Web service must have a 2-second response time" is better as this can be measured. "The Web service must provide accurate responses under 2 seconds on a 24x7 basis with up to 500 concurrent users" is even better. Having measurable requirements in place allows developers to work toward a known goal. Defining boundaries like "up to 500 users" gives them scope. Without these boundaries, there is no easy goal and code is likely to be over or underdeveloped.

Likewise, testers must know what functional and performance user requirements have been defined. Developers work on interpreting the user requirements to generate code. Test strategy should be done concurrently to ensure that testers work from the same requirements documents used by development. If test strategy is based on developed code as opposed to initial requirements, the resulting end product may be based on a single individual's misinterpretation. By creating tests and testing strategies based on the original set of user requirements, at least one more person interprets the project's needs. By working from the same initial requirements, if differences in requirement interpretation arise, they can be resolved early in the process as opposed to when development is near completion. In this way, the testing group acts as an early check and balance on the software development process.

Once service-level agreements or other quality of service contracts have been established, both parties (provider and consumer) need to agree on what measurements are to be taken and how, what metrics will be reported, what success criteria will be used as a measure of service quality, and what ramifications exist should the provider fail to deliver the agreed upon level of quality. Following deployment of the developed Web service(s), both parties have a vested interest in monitoring the ongoing delivery of the functionality. Web service (WS) testing and monitoring must cover the agreed upon metrics, which should include:

• Availability (accessibility): The applications must be running and reachable.
• Performance (scalability): Must respond to requests within acceptable time limits for expected consumer load.
• Accuracy (functionality): Must be accurate regardless of load.
• Standards compliance (interoperability): Must work with all potential client types (SOAP stacks).
• Integration: Must work with all other services and applications within the enterprise.

Managing the Quality Process
Managing the flow of development and testing from inception through deployment requires a great deal of communication. Requirements must be disseminated, business risks assessed, tests defined, results published, and any number of others. Each group and each individual within a given project have a different input into the system and each of these needs something back. For example, developers need the requirements, assumptions, limitations, etc., initially defined in order to code effectively. In return, they provide testable code to quality assurance. They may also provide knowledge in terms of problems encountered, expected problem areas, or other pieces of useful data. In the best scenario, they also provide test client code, scripts, and other unit-level intellectual property thus allowing QA to work more quickly and effectively than they would if starting from scratch.

Throughout the life cycle, different individuals provide data and extract information. Business analysts may input requirements and business risks and report on requirement test status. Project managers input priorities and look at current coverage, status, readiness, and other information to make the call as to when to go live. Even third-party auditors may be involved to ensure compliance with regulations, etc. A collaborative effort must be in place to ensure that everyone gets the information most relevant to them, in a manner that they can understand, while also allowing them to provide relevant data themselves.

Initial Testing
Unit (Component) Testing
As Web services are generally remote services that provide distributed computational functionality, they do not usually have GUIs from which to test. Developers have several options open to them to test the code they write: create test clients by hand, develop test clients with an Integrated Development Environment plug-in, use dynamically generated clients, or employ model (WSDL) based tools.

Regardless of the testing method, developers are testing the core functionality of the pieces of code they create, ensuring that the Web service does what it is supposed to do as a component. This may or may not include concurrency testing to ensure that multiple users can access the Web service simultaneously.

As coding and unit testing progress, the code, test results, and test clients need to be organized and passed on to the next stage. QA can use this intellectual property to shorten its ramp-up time prior to the next stage of testing. Development should also pass any other relevant information as stated earlier.

Functional Verification
Testing that a Web service or integrated set of Web services functions properly as a whole is generally the next step. The initial requirements lay out what the service should do. Testing that the Web services provide accurate responses to various input data sets ensures that the code is acceptable.

If test clients exist from previous unit tests, they can be reused and potentially modified to step through multiple iterations, testing both positive (works right when proper data is supplied) and negative (generates errors properly when conditions dictate) transactions. These tests should be scheduled to coincide with business risk and project priority as defined appropriately in the earliest stages of the project. Results of these tests then provide an input to the decision-making process.

Issues that arise during testing need to be evaluated with regard to several factors. These include questions like: Do the initial requirements need to be modified? Does any resulting code change invalidate testing that has already occurred? Should this issue be included in future regression testing? And finally, is the developer getting relevant information to help determine the issue's root cause and thus facilitate resolution?

Most of these questions can be resolved through the use of formal processes in conjunction with full-featured test management and defect tracking mechanisms. The necessary individuals can assess the effects of code and requirement changes on testing, schedules, resources, etc. The final question is best answered by reuse of initial unit testing technology. If the developer is receiving defect information using the same tool, script, or code that was used initially, the developer's analysis of that information will be quicker due to his or her familiarity with result formats.

Compliancy Testing
Web services are intended to be generic and accessible by any standards-based Web services client. In theory, a Java-based Web service client can access a .NET-based Web service and vice-versa. This, however, is not always the case as there are minor differences between the .NET SOAP stack and the Apache (Java) SOAP stack.

To ensure that a Web service is compliant, tests should be conducted using clients of all anticipated client types. For internal Web services, an organization may be certain that only one flavor of client will access a given Web service, eliminating this need. In all other cases, the unit tests above should be initiated from browsers, Java clients, and .NET clients using various browsers, JDKs, and .NET Frameworks to ensure that the Web service will interact accurately with any and all service request types.

Concurrency Testing
Once functional verification has been completed for single transaction sets, concurrency testing needs to be done. Since their very nature is distribution of business logic, Web services are accessible to numerous consumers. Nearly all Web services can be accessed by more than one consumer at a time. Concurrency testing ensures that the Web service works properly with a small number of concurrent users.

This testing is normally done as a precursor to load or performance testing. There is no point in designing a load test for one or more Web services if they can't handle two simultaneous sessions. Concurrency is simply a small test of a Web service that can be performed manually, through unit tests run concurrently or through small-scale tests of a load testing package. If the performance testing application is reusing code or scripts from earlier testing, using this mechanism makes good sense.

Scalability Testing
Once a development group knows that their Web services can be accessed concurrently, load testing can begin. The purpose of scalability tests is to ensure proper behavior of the Web service under expected loads.

Scalability tests come in a variety of flavors and each is designed with a slightly different goal. Load testing scenarios include:

• Increasing workload: Tests start with a small number of concurrent Web service calls and systematically raise the number during the test duration. This test type identifies bottlenecks and determines breaking points.
• Uptime: Tests provide a steady load on the system for an extended period of time. This test type is used to search for things like memory leaks that may only appear after hours or days of service usage.
• All-Day scenarios: Tests mix all the various Web service function call transactions in varying numbers throughout an expected normal time period. These tests are usually the final real-world scenario to ensure acceptable performance as measured by the defined service levels.
• What If testing: Testers may play with any number of unexpected or worst-case scenarios:
  - What if all potential consumers make the same function call simultaneously?
  - What if consumers make Web Service calls at varying or unforeseen connection speeds?
  - What if consumers make calls in an illogical order?
  - What if consumers make requests from different client SOAP stack types?

This test type is used to predict the less predictable issues that can arise when new applications are deployed to large sets of consumers. These tests are slightly less relevant for internal services, but are still of value.

In all cases, performance testing should also include some level of functional verification. Many complex applications create errors that occur sporadically under load conditions. If all Web service calls are not analyzed for accuracy, these failed functions may be missed. They are exactly the sort of errors that load tests are designed to generate, therefore it only makes sense to monitor all traffic for errors.

When errors or bottlenecks are uncovered, development needs to get relevant data like exception information, logs, communication headers, etc., to help determine what happened. Once again, reuse of technology allows development to receive this information in a familiar format.

Production Monitoring
Most of the industry literature about service-level agreements falls into the production monitoring space. Once a Web service has been deployed, people understand that it should be monitored to ensure the previously set thresholds are achieved in terms of availability, performance, accuracy, and interoperability as discussed previously. In most operational cases, however, the group monitoring the Web services is not the same as the group who developed and tested it. This disconnect between application development and operations means that the production monitoring must be created from scratch, often with little or no information learned during the development and testing process.

Once again, technology reuse can be a large asset. By providing operational personnel with the same technology that was used to test the application before, time and effort can be saved. More importantly, threshold information, likely problem areas, and other information learned through testing can be transmitted to the production world, giving them a head start on what and how to test. In the event of a failure in production, the reused technology once again means that developers see the information in a familiar format. As always, this shortens the analysis cycle, thereby reducing time to repair and keeping downtime to a minimum.

The Integrated Quality Management Life Cycle
In short, knowledge sharing and technology reuse are core components of ensuring the quality delivery of Web services. By integrating people through good practices and formal processes, each and every individual has the information needed to perform tasks in an optimal fashion. By reusing technology from initial unit testing on the developer's desktop through functional and scalability testing all the way through to production monitoring and back again, testing cycles are not only shortened, but information flow in both directions is enhanced.

Putting all of the pieces together in this fashion keeps Web service development and deployment quality at its highest. Business runs smoothly with critical functionality distributed through these high quality Web services. Consumers get the correct and timely responses to function calls each time, all the time. That's why you started using Web services in the first place, isn't it?

Summary
Today's world of interconnected business entities requires standard computing mechanisms. Web services allow well-defined remote functionality for access by internal and external consumers to be easily developed and deployed. Since various Web services can integrate across numerous providers, quality of service must be maintained at a high level. Managing the quality life cycle from design to delivery ensures that businesses can continue to rely on these deployed services now and in the future. Employing collaborative, integrated, and reusable technology ensures that high levels of quality are achieved and maintained optimally throughout the process.