Groovy, Low Budget Web Application Testing

Marty Kube
Principal Consultant, Beaver Creek Consulting

Introduction

This posting presents a lightweight test harness for exercising a web application.  The advantages of have test cases while coding are clear.  I've previously written about quickly setting up a regression test under limited time constraints.   Here, I show a concrete example built in the low budget spirit.

This type of test harness is useful if you find yourself:

  • On a new project and having to implement a new feature or bug fix,
  • Worried about unknown side effects that will break the system,
  • Bummed because there are no test cases, and
  • Under pressure to get things done quickly and correctly.

I've used the Groovy language to create a test harness for black box testing.  The test harness reads a list of GET or POST requests from a file, fires them off against a server, collects the returned HTML, and compares the results against a known good result.

Options

Writing your own test harness doesn't make sense in every case.  It is worth considering if you want to quickly construct a few simple test and you are the only user of the tool.  If your testing needs are more complex or you expect the tool to be used by a number of people, it might better to use an available testing tool.

This question might be framed as a Build vs. Buy decision.  I've always thought that Buy was the right solution, especially when "Buy" means download a free open source package.  On the other hand, building your own test harness can give you much more control.  I find myself going with the build route when the feature set is small enough to outweigh the learning curve of adopting new software.  The use case here is quick development of a custom testing tool.  I'm comfortable recommending writing your own test harness in this case.

A Web Application Test Harness

I've written a simple web application that I'll use as the application under test.  The test application is available here.  GET request return a page with a generation time stamp and POST request also add the request parameters in the returned page.  The following snippet is part of the returned page for one of the example test cases from this article:
This is a test page for the article low budget regression testing.
Query parameter: groovy: rocks!
Query parameter: Testing: Is good for you
Page generated at: Wed Jan 16 14:48:24 2008
The output of the application has features of a typical application that a test harness has to deal with; some content that changes on each returned page (the time stamp) that needs to be ignored while evaluating the quality of the results.

Specifying the Test Cases

The test cases are stored in an XML file.  An example of a test case file is:

001: <?xml version="1.0"?>
002: <!DOCTYPE test-cases [
003:     <!ELEMENT test-cases (test-case+)>
004:     <!ELEMENT test-case (post-param*, filter*)>
005:     <!ELEMENT post-param (param-name, param-value)>
006:     <!ELEMENT param-name (#PCDATA)>
007:     <!ELEMENT param-value (#PCDATA)>
008:     <!ELEMENT filter (#PCDATA)>
009:     <!ATTLIST test-case method (GET|POST) "GET">
010:     <!ATTLIST test-case url CDATA #REQUIRED>
011:     <!ATTLIST test-case name CDATA #REQUIRED>
012: ]>
013:
014: <test-cases>
015:
016:     <test-case name="Case1" method="GET"
017:         url="http://beavercreekconsulting.com/cgi/low-budget-test.pl">
018:         <filter>Page generated at: \w{3} \w{3} \d+ \d+:\d+:\d+ \d+</filter>
019:     </test-case>
020:
021:     <test-case name="Case2" method="POST"
022:         url="http://beavercreekconsulting.com/cgi/low-budget-test.pl">
023:         <post-param>
024:             <param-name>groovy</param-name>
025:             <param-value>rocks!</param-value>
026:         </post-param>
027:         <post-param>
028:             <param-name>Testing</param-name>
029:             <param-value>Is good for you</param-value>
030:         </post-param>
031:         <filter>(?s)\&lt;p\&gt;Query parameter:[^&lt;]+&lt;\/p&gt;</filter>
032:         <filter>Page generated at: \w{3} \w{3} \d+ \d+:\d+:\d+ \d+</filter>
033:     </test-case>
034:
035: </test-cases>

I've added a DTD so that the XML parser can let me know when I get off track.  The file contains list of test cases.  Each test case has a name and is either a GET or POST to the specified URL.  For POST, the post parameters are a list of name/value pairs.  The filter element is a list of regular expression filters that are applied to the actual and expected results prior to comparison.

Specifying a regex in an XML file can involve a bit of escaping.  The filter on line 31 after XML parsing is:

(?s)\<p\>Query parameter:[^<]+<\/p>
Which is intended to match a HTML paragraph, which possibly spans multiple lines, starting with the literal 'Query parameter:'.

The Test Harness

The console output while running the test harness is:

Reading test cases...
2 case(s) to process
Fetching: http://beavercreekconsulting.com/cgi/low-budget-test.pl... 200 OK
Fetching: http://beavercreekconsulting.com/cgi/low-budget-test.pl... 200 OK
Filtering Case1
Filtering Case2
Comparing...
Case1: PASS
Case2: PASS

The code for the test harness is show below.

The import statements (lines 1-6) bring in the excellent Jakarta Commons HTTP client to handle HTTP communications with the server.  The class TestCase (lines 11-16) holds the information read from the test case file.  Each instance of the class represent one test case. 

The next step is to read the test case file and populate TestCase instances (lines 18-35).  The request is made to the server and the response piped to the actuals file (lines 46-64).  The actuals and baseline (known good results) files are filtered (lines 67-73) and compared (lines 75-96).  A pass/fail status is emitted based on the differences between the filtered actuals and baseline file.

Conclusions

Setting up a test harness doesn't have to be huge effort.  It is better to get something simple in place than to code without test cases.  I've shown here how a simple script can be used to get your testing program up and running quickly.

001: import org.apache.commons.httpclient.HttpClient;
002: import org.apache.commons.httpclient.NameValuePair;
003: import org.apache.commons.httpclient.UsernamePasswordCredentials;
004: import org.apache.commons.httpclient.auth.AuthScope;
005: import org.apache.commons.httpclient.methods.GetMethod;
006: import org.apache.commons.httpclient.methods.PostMethod;
007: import org.xml.sax.ErrorHandler;
008: import org.xml.sax.SAXParseException;
009:
010: // Test case
011: class TestCase {
012:     def name, method,filters = [];
013:     String toString() {
014:         return "[TestsCase:(${name}, ${method}, ${filters}]";
015:     }
016: }
017:
018: // Read test cases
019: println "Reading test cases..."
020: def testCaseList = [], parser = new XmlParser(true, false);
021: parser.setErrorHandler(new TheTerminator());
022: parser.parse(new File('test-cases.xml')).'test-case'.each() { testCase ->
023:     def inputTestCase = new TestCase();
024:     inputTestCase.name = testCase.'@name';
025:     inputTestCase.method = ((testCase.'@method' == 'POST')
026:             ? new PostMethod(testCase.'@url')  : new GetMethod(testCase.'@url'));
027:     testCase.'post-param'.each() { param ->
028:         inputTestCase.method.addParameter(
029:             new NameValuePair(param.'param-name'.text(), param.'param-value'.text()));
030:     }
031:     testCase.filter.each() { filter ->
032:         inputTestCase.filters << filter.text();
033:     }
034:     testCaseList << inputTestCase;
035: }
036: println "${testCaseList.size()} case(s) to process"
037:
038: // Fetch pages into actuals
039: client = new HttpClient();
040:
041: // If you're behind a proxy firewall, this snippet is handy to have
042: //client.getState().setProxyCredentials(AuthScope.ANY,
043: //    new UsernamePasswordCredentials("user", "password"));
044: //client.getHostConfiguration().setProxy("proxy-server", proxy-port);
045:
046: testCaseList.each() { testcase ->
047:     def method = testcase.method, httpReader, actuals;
048:     try {
049:         print "Fetching: ${method.getURI()}..."
050:         client.executeMethod(method);
051:         println " ${method.getStatusCode()} ${method.getStatusText()}"
052:         if(method.getStatusCode() != 200) throw new RuntimeException(
053:             "Failed to fetch ${method.getURI()}"
054:         );
055:       actuals = new FileWriter("${testcase.name}-actuals.txt");
056:         new InputStreamReader(method.getResponseBodyAsStream()).eachLine() {
057:             actuals << it << "\n";
058:         }
059:     } finally {
060:         method?.releaseConnection();
061:         httpReader?.close();
062:         actuals?.close();
063:     }
064: }
065:
066: // filter actuals and baseline
067: testCaseList.each() { testcase ->
068:     println "Filtering ${testcase.name}";
069:     filterFile("${testcase.name}-actuals.txt",
070:             "${testcase.name}-actuals-filtered.txt", testcase.filters);
071:     filterFile("${testcase.name}-baseline.txt",
072:             "${testcase.name}-baseline-filtered.txt", testcase.filters);
073: }
074:
075: // compare actual and expected
076: println 'Comparing...';
077: def diff = 'cmd /c \\cygwin\\bin\\diff.exe -c ';
078: testCaseList.each() { testcase ->
079:     print("${testcase.name}: ");
080:     cmd = """
081:         ${diff}
082:         ${testcase.name}-baseline-filtered.txt
083:         ${testcase.name}-actuals-filtered.txt
084:     """.execute();
085:     out = cmd.text;
086:     err = cmd.err.text;
087:     cmd.waitFor();
088:     switch(cmd.exitValue()) {
089:         case 0: // no diff
090:             println 'PASS'; break;
091:         case 1: // diffs found
092:             println 'FAIL'; println "DIFFS:\n${out}"; break;
093:         case 2: // diff error
094:             println 'FAIL'; println "ERROR:\n${err}"; break;       
095:     }
096: }
097:
098: // apply the test case filters
099: def filterFile(fromFilename, toFilename, patternList) {
100:     def fixedUp = new FileReader(fromFilename).getText();
101:     patternList.each() { pattern ->
102:         def matcher = (fixedUp =~ pattern);
103:         fixedUp = matcher.replaceAll('__FILTERED__');
104:     }
105:     def to = new FileWriter(toFilename);
106:     to << fixedUp;
107:     to.close();
108: }
109:
110: // XML parser validation callbacks
111: class TheTerminator implements ErrorHandler {
112:         public void error(SAXParseException exception) {terminate(exception)}
113:         public void fatalError(SAXParseException exception) {terminate(exception)}
114:         public void warning(SAXParseException exception) {terminate(exception)}
115:         void terminate(SAXParseException exception) {
116:             println("Parse error ${exception}")
117:             System.exit(1);
118:         }
119: }

About the author

  1. Marty Kube ( martykube@beavercreekconsulting.com ) is a Principal Consultant with Beaver Creek Consulting Corp.  Marty has 10 years of professional experience as a Software Developer and specializes in developing finance and accounting solutions.  Marty is a Sun Certified Java Programmer with expertise in J2EE and database systems.  He holds a Doctorate in Chemical Engineering from the University of Virginia.

Last Update: 20080118