Use Validators and Load Generators to Test Your Web Applications continued...

Use a Load Generator
to Test Your Server Software

Stress test your web applications to find performance problems.
Links to load generators for testing HTTP, HTTPS and other protocols.

Mike Crawford

Mike Crawford
Consulting Software Engineer
mike@soggywizards.com

Copyright © 2001, 2012 Michael D. Crawford.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 License.

Use a Load Generator
to Test Your Server Software

Links checked and updated May 2, 2012.

Many, many new stress-testing packages have been released, and techniques have been developed since I first wrote this article back in 2001.

I can easily find many of them on my own, and will link them soon, but if you publish, or even just know about any manner of load generator whose license complies with either the The Free Software Definition or The Open Source Definition, please do send the link to your package to mike@soggywizards.com

I got my first look at a load generator when I was working at Octel Communications, a company that built multiprocessor computers with a custom-developed real-time operating system for use in voice mail. These were impressive machines; the larger models basically consisted of refrigerator-sized card cages with dozens and dozens of 80386 chips with half a cubic meter of hard disks in the base for storage of the digitized voice messages.

I often worked through the night in my job as a network administrator to take down the Sun servers for maintainence, and I noticed that in the test lab outside the computer room at night there was a machine in a rack mount with hundreds of "silver satin" telephone patch cords coming out of it making a cacophony of touch-tone beeps.

I asked my manager, Karen Coates, what this strange device was and she told me "It's a load generator". It had the job of testing our messaging software and hardware by placing an endlessly repeating series of simultaneous phone calls to a system under test and leaving machine-generated voice mail.

Wanna do lunch today?

I think load generators are likely in more widespread use in the real-time and embedded software areas, but they are available for testing web and other internet applications.

Note that I entitled this section "Use a Load Generator to Test Your Server Software".

You might say that what we need to stress test is the hardware, for example to predict how powerful a server one needs to buy to handle a certain number of simultaneous users.

That's a valid use for a load generator, but I suggest that what you really need to concentrate on is the performance of your software. Performance problems can be insidious and hard to fix in a completed application; it is important for each programmer to do load testing on a daily basis during development, and for extensive stress testing during each step of a formal quality assurance process.

If you don't, you might find that your half-million dollar server falls over after you take your site live and just a few dozen people log in.

Let's say we get extra payoff in hardware performance out of proportion to the extra cash we spend; suppose paying twice as much for a server, hundred of megabytes more memory or faster RAID storage for your database will increase performance by a factor of four.

That's not a bad investment. What is a bad investment is that one lazy, inexperienced or uneducated programmer, or an otherwise good programmer who is ill-informed, working with a poorly designed overall architecture or not communicating effectively with his coworkers can take that two-fold investment in server hardware and reduce the performance of your server a hundredfold in just a few lines of bad code - code that will be devilishly hard to find once your application is completed.

Performance problems may not be contained in one small section of code but be spread throughout your program, and they could be caused by the way disparate parts of your program interact with each other - or by interactions between different server processes in a system.

If you are running more than one process (as is typically the case) or using a multithreaded implementation, you need to worry about race conditions, deadlocks, or performance degradation caused simply by holding a lock longer than necessary. You may not notice any of these problems at all unless you stress test.

Another significant problem is memory leaks. Having a memory leak in your web application may slow your response due to violent paging and eventually run your server out of virtual memory backing store and crash your entire server, but you may not find out that you even have a memory leak in your application because manual testing of your pages does not result in memory allocation growing fast or far enough to be noticed.

It is popular to write web applications these days in garbage-collected languages like Java, Python or Perl, and you might object that a garbage collected application is inherently free of memory leaks. While it is true that this is a common claim of marketing literature for some of these languages, it is simply not true. I've experienced plenty of examples of memory leaks in garbage-collected applications written in these and other languages such as Smalltalk in the course of my career, and I've heard of far more.

Under garbage collection, an object will not be freed if some other object holds a reference to it. All you need to do to have a memory leak in a garbage collected application is to be careless about keeping unwanted references around.

A single reference to the root node of a DOMTree in a Document Object Model application can hold megabytes of XML data in memory - permanently so. You may not be able to find where this reference is kept no matter how hard you search your source code for it. You may not even know it's there because you never bother to look.

It's as simple as that. Actually fixing or preventing performance problems or memory leaks in your web applications is beyond the scope of this article; the first step is to detect them, and for that we can use a load generator.

Proposal for Validation During Stress Testing

I'd like to suggest that someone with the inclination to do the work should combine one of the available validators with a load generator, so that all (or perhaps just a sample) of the documents downloaded from the machine under test are validated.

Because of the extra work required to validate, your load generator / validator would need to run on a more powerful machine than the server under test, or you would need to run the attack with several load generators working simultaneously.

This would serve an important purpose. While you can use a validator to check that your pages are correct in basic use, and you can use a load generator to ensure your server system can withstand the expected user load, it is hard to tell that the pages being generated by a stressed server are actually what you want.

If bugs due to stressful conditions cause corrupt documents, then such a validating load generator is likely to find it.

Rigorous HTML Validation is quite resource intensive. If you validate every document completely immediately upon receiving it from the servver under load, you'll find that the load generating machine itself is overloaded. There are lots of ways to address this problem:

You can leave your tip in the jar on the way out. Thanks folks, you've been swell. I'll be playing all week.

The Available Load Generators

First, an important warning:

Using a load generator can cause signicantly large amounts of network traffic and CPU load on your server. You are advised to avoid using a load generator on a live site over the Internet as you may incur costly bills for bandwidth usage. If your server is shared with other sites, as is the case with database hosting services, the administrators may kick you off for hogging the CPU.

I suggest that even on an intranet you isolate the load generation machine and the system under test by using an ethernet switch or router to put them on a private network segment, otherwise your coworkers may curse you for interfering with their ping times whenever you turn on your test tool. Or just unplug their connection to the rest of the office.

For day-to-day use by an individual developer, it will probably do to put the load generation software and the application under test on a single workstation, so you don't generate network traffic at all or have to spend a lot of money on testing hardware.

You can also try putting a load generator on your laptop as I suggest in my discussion of installing the W3C validator source code. That way maybe you can grab a latte and listen to some nice jazz in a cafe while you develop your web application.

I list the available load generators for which source code is available at the time of this writing. I found the load generators by using the following searches on Freecode:

Deluge - Download

Web site testing tool with included proxy server for recording playback scripts, and log evaluator for generating result statistics.

Hammerhead - Download

Hammerhead is a web site coverage, HTTP load generator, HTTP benchmarking, and stress testing tool. It has been designed to emulate multiple users from multiple IP addresses at maximum speed

ISIC -- IP Stack Integrity Checker - Download

Network stack tester. It sends pseudo-random packets to the selected protocol to test the stability of the drivers and to check for leaks in firewalls. It now supports IPv6, and can generate raw ethernet packets to test hardware implementations.

Even if you are using top-quality hardware, with stable, highly regarded software whose vendors test rigorously, you have no real way of knowing whether your particular installation of your network server really is reliable, or is even configured correctly. There are all kinds of things that can go wrong that hardware and software vendors have no control over. Therefore you would do well to use a tool like ISIC before putting any network server into production.

And I Quote: ISIC may break shit, melt your network, knock out your firewall, or singe the fur off your cat.

openload

Load testing of web applications.

Ptester - Peter's HTTP Benchmarking Tool - Download

Simple HTTP benchmarking and stress-testing tool

Rjobd Toolbox - Download?

Framework for reliably controlling the execution of processes on a networks of computers.

See author Mike Werner's paper Evolving the Resilience of an Operating System. Where?

The Rjobd Toolbox was at one time Open Source, but appears not to be anymore. I'm not yet able to find the last release of its GNU General Public License source code, but I expect I can turn it up. So far it looks like the last Open Source release was Version 1.0 in 1994; if you know where I can find any version at all of the Rjobd source, or can even give me a clue as to where next to look, please email me at mike@soggywizards.com

From Rjobd's page:

The Rjobd Toolkit is the enabling technology for the Chaos Resilience Testing technique. This technique exploits elements of complexity theory to uncover an addional class of reliability problems that are otherwise delivered to the customer base.

But note from the no longer available Freshmeat page:

Take note that installing the current version will more or less completely compromise system security.

Siege - Download - Manual

HTTP and HTTPS load testing and benchmarking utility. Supports basic authentication, cookies, and a configurable number of simulated web browsers.

SLAMD - Download

Multiprotocol stress-testing and performance analysis tool. Supports such protocols as HTTP, LDAP, IMAP, POP and SMTP and can query a relational database via JDBC. Java-based API for custom work loads, scripting engine with recording and playback for developing tests.

smpop - Download?

POP (Post Office Protocol) stress tester that simply pulls out a lot of email from a server given a list of usernames and passwords

smpop seems to have entirely disappeared. I don't think that it was withdrawn from Open Source as Rjobd was, but that it's author simply abandoned it and took down its entire web site.

The last release I find any mention of it was it's 2.0-release version from 2002, with its site going offline in 2004. Its source code was contained in two apparently identical archives: smpop-0.2-release.tar.gz and smpop-current.tar.gz

Again if you can yield any insight at all as to how or where I can find smpop's source code, please do email mike@soggywizards.com

SSL Stress Client - Download Coming Soon - README

Small program for stress testing SSL servers. It is also capable of testing static HTTP web pages.

This project has also been abandoned by its developer, but I was able to find the source!

I do not feel safe posting the source code just yet as there is some chance that the source I was able to find has been... modified... from its original. Lest hilarity ensue when I pass it on to you, I'll be having a good look over the code to make sure no Easter Eggs are found within.

If you want to take the risk of running the code yourself, before I've vetted it, it's easily found on the Web. The tarball I found is:

Please don't feel that because I provide hashes of the source tarball that I downloaded from some random h4x0r site, that I'm claiming it's not full of malware!

It has the Four Clause BSD license, that is, the original BSD license with the troublesome advertising clause. That's not at all a problem for those who just wish to use SSL Stress Client, but there are all kinds of reasons that makes this code troublesome for those who wish to create derivative works of it. If you work with BSD-licensed code in any way, educate yourself on the issues around the Four Clause and Three Clause BSD licenses, then if you possibly have a choice, use the Three Clause one.

Tsung

Load test tool for HTTP, PostgreSQL, Jabber and others.

Next Page Previous Page Contents All Webmaster and Web Design Tips