Success Story e-TEST suite

Countrywide Home Loans

Case Study: Countrywide Home Loans

"The ease-of-use and powerful reporting of RSW Software’s e-TEST™ suite has allowed us to quickly evaluate changes to our applications, increasing our overall performance dramatically -- a factor of two to three times faster than when we started. In addition, unlike other testing software, RSW’s e-TEST™ suite minimizes the overall expense associated with load testing, allowing us to simulate a realistic number of virtual users on just a couple of machines…. Ultimately, the e-TEST™ suite will help us position ourselves competitively for future growth."

Background

Recognized as an innovator in developing new financial products and services, Countrywide is the nation’s leading independent residential mortgage lender and servicer. By combining creative new ideas with a philosophy that puts the customer first, Countrywide has been able to achieve its’ goal of providing faster and more efficient service. Part of Countrywide’s success has rested upon their mission to achieve their goals through the use of technology.

With a desire to utilize state of the art technology while offering their customers the highest level of service, Countrywide embarked on its Internet development program four years ago. Since that time, Countrywide has expanded its offerings to include a variety of home financing and financial services. Based on the success of these early programs, Countrywide's current plan is to deploy as many as 1000 new applications over the next two years as part of their Internet and Intranet sites.

As they planned to move all their business-critical applications to the Web, Countrywide was faced with a challenge: preventing the creation of Web applications that would be difficult to develop and maintain, that wouldn’t scale to the growth needs of the company, and wouldn't perform as needed.

To meet this challenge, Countrywide's engineers came up with a technical approach to support their goal of becoming a Web-centric company. They decided to develop a standard infrastructure that was flexible enough to support the majority of their applications while at the same time maintaining overall site performance and quality. They selected the Sun-NetDynamics application server as a universal middle-tier that would offer application-builders simple connectivity to any of the company's database servers along with a rapid application development environment.

Testing Challenges

In order to ensure that their new infrastructure would support such a large number of applications, Countrywide recognized the need to validate the scalability of their overall system architecture early on - even before the first application was built. They wanted to identify and eliminate performance bottlenecks before they wrote their first line of application code. Then, as applications were developed, they needed a fast way to test each of these applications for functionality and performance. Countrywide had used automated testing tools before, but discovered that there was often more work involved in learning the software package and writing and maintaining the test programs than there was to building the application itself. Countrywide needed a practical solution for automated Web application testing that could be used by people across their entire organization including developers, QA professionals, and Business Analysts. They selected the e-TEST™ suite from RSW Software as the only tool set that could meet their needs, choosing the product because of its overall ease of use and its ability to emulate a large number of concurrent users using just a handful of machines.

Testing Process

Countrywide needed to test both the functionality and performance/scalability of their NetDynamics application servers. They decided to first establish a performance "baseline" of their existing applications so they would have a frame of reference for their new infrastructure. Countrywide then used e-LOAD™ to run a series of load tests on their existing applications, simulating a series of heavy loading conditions on their servers. Because Countrywide did not yet have their new applications in place, their engineers wrote a series of simple, generic browser-based applications in order to be able create similar loads on the new system and to compare them against the baseline.

Next, using e-LOAD™, they simulated a scenario of 250 concurrent users performing a mix of operations simultaneously (both with and without error conditions) in order to test the scalability of various pieces of the system. Then they proceeded to perform a series of "what-if" experiments: using e-LOAD™ as the reference load they changed different aspects of the system, while monitoring the Web servers, application servers, database servers, and network. After each change to the system, they used RSW’s e-TESTER™ to ensure that the latest build still functioned correctly, and e-LOAD™ to measure the effect of the modification on overall performance.

Results

The initial results of the load test were dreadful - far below the high standard of performance that Countrywide had set for its applications. The second set of experiments (using e-LOAD™) enabled Countrywide to isolate a number of bottlenecks in the system, after which they made various global changes as well as a number of fine-tuning adjustments. The changes to the system that had the greatest impact included: repartitioning applications to group similar types on the same servers, upgrading some of the busiest network segments, and co-locating the database servers and application servers.

After generating their results, the Countrywide team was highly impressed with the detail and clarity of the reports generated from e-LOAD™, and was able to see dramatic changes with each test they ran. With the configuration changes and enhancements they made to the system, Countrywide witnessed a two to three-fold improvement in response time over the initial test, realizing that application testing can have a tremendous impact on the bottom line.