Erlang Central

Difference between revisions of "Testing with Test Helper"

From ErlangCentral Wiki

m
m
Line 7: Line 7:
 
For this you can use the erlang test server. It's home page is http://www.erlang.org/project/test_server/index.html. It is quite an involved process getting it to work, as you can find on [http://www.erlang.org/project/test_server/README this page].
 
For this you can use the erlang test server. It's home page is http://www.erlang.org/project/test_server/index.html. It is quite an involved process getting it to work, as you can find on [http://www.erlang.org/project/test_server/README this page].
  
In order to speed up the testing process and to fit in with a philosophy of test as you go, we created [[File:testhelper.erl]]. It makes it easy to build test suites that are compatible with the [[Test Server]], but still allows you to run your tests in [[Emacs]].
+
In order to speed up the testing process and to fit in with a philosophy of test as you go, we created <tt>testhelper.erl</tt>. It makes it easy to build test suites that are compatible with the [[Test Server]], but still allows you to run your tests in [[Emacs]]. It doesn't support all the features of the Test Server, but does make it easier to test and develop on the go.
  
 
==Writing a test suite==
 
==Writing a test suite==
Line 18: Line 18:
  
 
===Generate the skeleton===
 
===Generate the skeleton===
In [[Emacs]] select Erlang|Skeletons|Erlang test suite TS Frontend:
+
1. In Emacs, create a file named <tt>simple_test_SUITE.erl</tt><br>
 +
2. In [[Emacs]] select Erlang|Skeletons|Erlang test suite TS Frontend:
  
 
[[Image:testhelper_step_1.png]]
 
[[Image:testhelper_step_1.png]]
Line 56: Line 57:
 
   ok.
 
   ok.
  
Next, compile the suite:
+
3. Next, compile the suite:
  
 
  (gdb_rdbms@wyemac)1> c("simple_test_SUITE",[debug_info]).
 
  (gdb_rdbms@wyemac)1> c("simple_test_SUITE",[debug_info]).
Line 81: Line 82:
 
  ok
 
  ok
  
At this stage you'll notice that testhelper indicated 1 tests passed, 0 failed. The test that passed was our test_case(...) created above. Of course it doesn't do anything yet.  
+
At this stage you'll notice that testhelper indicated 1 tests passed, 0 failed. The test that passed was our <tt>test_case(...)</tt> created above. Of course it doesn't do anything yet.  
  
 
Let's make it fail. Change test_case(...) as follows:
 
Let's make it fail. Change test_case(...) as follows:
Line 97: Line 98:
 
   ok.
 
   ok.
  
The test simply creates two atoms, one name My1stAtom, the other My2ndAtom. In the third line, we make an assumption that these two atoms are the same by trying to match the two variables.
+
The test simply creates two atoms, one named <tt>My1stAtom</tt>, the other <tt>My2ndAtom</tt>. In the third line, we make an assumption that these two atoms are the same by trying to match the two variables.
 
Lets test this. Run testhelper again:
 
Lets test this. Run testhelper again:
  
Line 156: Line 157:
 
This time, testhelper tells us that test_case failed again, but on line 33 of the code. Much better.
 
This time, testhelper tells us that test_case failed again, but on line 33 of the code. Much better.
  
 +
===Configuration Cases===
 +
Testhelper respects [http://www.erlang.org/project/test_server/html/test_spec_chapter.html#3.4 configuration cases]. It treats each configuration case the same way [[Test Server]] does and passes state to the nested cases.
 +
 +
Configuration cases are basically a mechanism that helps with setup and the teardown of tests. They have the ability of changing the configuration passed to the tests embedded inside them.
 +
 +
Here is an example of a configuration case with a nested test case:
 +
 +
configuration_start(doc) ->
 +
  ["Start of configuration case"];
 +
configuration_start(suite) ->
 +
  [];
 +
configuration_start(Config) when is_list(Config) ->
 +
  io:format("Configuration test case starts...\n"),
 +
  Config.
 +
 +
test_case2(doc) ->
 +
  ["A 2nd Test Case"];
 +
 +
test_case2(suite) ->
 +
  [];
 +
 +
test_case2(Config) when is_list(Config) ->
 +
  io:format("This is test 2!\n"),
 +
  ok.
 +
 +
configuration_stop(doc) ->
 +
  ["Start of configuration case"];
 +
configuration_stop(suite) ->
 +
  [];
 +
configuration_stop(Config) when is_list(Config) ->
 +
  io:format("Configuration test case ends!\n"),
 +
  Config.
 +
 +
We also had to change the all(...) test case as follows:
 +
 +
all(doc) ->
 +
  ["This test links to all tests in this suite"];
 +
 +
all(suite) ->
 +
  [test_case,
 +
    {conf,
 +
    configuration_start,[test_case_2],
 +
    configuration_stop}].
 +
 +
Note how we embedded our new test_case_2 inside a configuration case.
 +
 +
Now compile and run it:
 +
 +
(gdb_rdbms@wyemac)18> testhelper:run(simple_test_SUITE,all).
 +
****************************************************
 +
Running simple_test_SUITE:init_per_suite (runtime version) Case all
 +
====================Start of Test test_case======================
 +
This is test 1!
 +
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 +
Testhelper catch error {badmatch,the_new_atom}
 +
[{simple_test_SUITE,test_case,1},
 +
  {testhelper,execute_test,3},
 +
  {timer,tc,3},
 +
  {testhelper,run_test2,4},
 +
  {testhelper,'-run_test/4-fun-0-',6},
 +
  {proc_lib,init_p,3}]
 +
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 +
====================End of Test test_case========================
 +
Starting Conf Case configuration_start...
 +
====================Start of Test configuration_start======================
 +
Configuration test case starts...
 +
====================End of Test configuration_start========================
 +
====================Start of Test test_case_2======================
 +
This is test 2!
 +
====================End of Test test_case_2========================
 +
====================Start of Test configuration_stop======================
 +
Configuration test case ends!
 +
====================End of Test configuration_stop========================
 +
Running simple_test_SUITE:end_per_suite (runtime version)
 +
****************************************************
 +
simple_test_SUITE:test_case                                                                        failed (501us) Line 37 []
 +
2 tests passed
 +
1 tests failed
 +
ok
 +
 +
==How it works==
 +
Testhelper runs each test in a separate process. This minimises the chances of test interfering with each other.
 +
 +
It doesn't print the test names that passed, as you can imaging printing out 40 tests that passed and one that failed you might just not be able to spot the broken one. On of course, our tests always works ;).
 +
 +
Testhelper skips any tests cases inside a configuration case if the start case failed, but will continue running the other tests if one nested one fails.
 
==Download==
 
==Download==
 
[http://www.patternmatched.com/download/testhelper.zip testhelper.zip]
 
[http://www.patternmatched.com/download/testhelper.zip testhelper.zip]

Revision as of 10:56, 2 September 2006

Contents

Author

Rudolph van Graan

Article

For this you can use the erlang test server. It's home page is http://www.erlang.org/project/test_server/index.html. It is quite an involved process getting it to work, as you can find on this page.

In order to speed up the testing process and to fit in with a philosophy of test as you go, we created testhelper.erl. It makes it easy to build test suites that are compatible with the Test Server, but still allows you to run your tests in Emacs. It doesn't support all the features of the Test Server, but does make it easier to test and develop on the go.

Writing a test suite

The basic steps are exactly the same as for Test Server. For details, look at Writing Test Suites

Here is the quick recipe:

  1. Generate the skeleton in Emacs
  2. Create a number of tests
  3. Run the test suite using testhelper

Generate the skeleton

1. In Emacs, create a file named simple_test_SUITE.erl
2. In Emacs select Erlang|Skeletons|Erlang test suite TS Frontend:

Testhelper step 1.png

This will fill in your current module with a basic skeleton. If we remove all the comments, the test should look like this:

-module(simple_test_SUITE).

-compile(export_all).
-include("test_server/include/test_server.hrl").

init_per_suite(Config) ->
  Config.

end_per_suite(_Config) ->
  ok.

init_per_testcase(_TestCase, Config) ->
  Config.

end_per_testcase(_TestCase, _Config) ->
  ok.

all(doc) -> 
  ["This test links to all tests in this suite"];

all(suite) -> 
  [test_case].

test_case(doc) -> 
  ["Describe the main purpose of test case"];

test_case(suite) -> 
  [];

test_case(Config) when is_list(Config) -> 
  ok.

3. Next, compile the suite:

(gdb_rdbms@wyemac)1> c("simple_test_SUITE",[debug_info]).
{ok,simple_test_SUITE}


Running all the tests in the new suite

In order to run our new test suite, we simply call testhelper like this:

(gdb_rdbms@wyemac)2> testhelper:run(simple_test_SUITE,all).
****************************************************
Running simple_test_SUITE:init_per_suite (runtime version) Case all 
DataDir = "/Users/rvg/svn/modules/common/test/simple_test_SUITE_data/" 
Config = [{data_dir,"./simple_test_SUITE_data/"},
          {priv_dir,"./simple_test_SUITE_priv/"}]


====================Start of Test test_case======================
====================End of Test test_case========================
Running simple_test_SUITE:end_per_suite (runtime version)
****************************************************
1 tests passed
0 tests failed
ok

At this stage you'll notice that testhelper indicated 1 tests passed, 0 failed. The test that passed was our test_case(...) created above. Of course it doesn't do anything yet.

Let's make it fail. Change test_case(...) as follows:

test_case(doc) -> 
  ["Describe the main purpose of test case"];

test_case(suite) -> 
  [];

test_case(Config) when is_list(Config) -> 
  My1stAtom = the_new_atom,
  My2ndAtom = the_other_atom,
  My2ndAtom = My1stAtom,
  ok.

The test simply creates two atoms, one named My1stAtom, the other My2ndAtom. In the third line, we make an assumption that these two atoms are the same by trying to match the two variables. Lets test this. Run testhelper again:

(gdb_rdbms@wyemac)4> testhelper:run(simple_test_SUITE,all).
****************************************************
Running simple_test_SUITE:init_per_suite (runtime version) Case all 
====================Start of Test test_case======================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Testhelper catch error {badmatch,the_new_atom}
[{simple_test_SUITE,test_case,1},
 {timer,tc,3},
 {testhelper,run_test2,4},
 {testhelper,'-run_test/4-fun-0-',6},
 {proc_lib,init_p,3}]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================End of Test test_case========================
Running simple_test_SUITE:end_per_suite (runtime version)
****************************************************
simple_test_SUITE:test_case                                                                         failed (84ms) (No Line) []
0 tests passed
1 tests failed
ok

This time, one test failed. You can see in the debug output that a bad match error occurred. The reason for this is that we tried to match two Atoms and they are just not created equal.

Unfortunately, we have to figure out where the error occurred (The third line of the third function clause), but in larger test suites, it becomes difficult to trace errors. Let us fix this. Change the code and recompile:

test_case(Config) when is_list(Config) -> 
  ?line My1stAtom = the_new_atom,
  ?line My2ndAtom = the_other_atom,
  ?line My2ndAtom = My1stAtom,
  ok.

The ?line macro is a mechanism that helps you find the error. Run testhelper again:

(gdb_rdbms@wyemac)6> testhelper:run(simple_test_SUITE,all).
****************************************************
====================Start of Test test_case======================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Testhelper catch error {badmatch,the_new_atom}
[{simple_test_SUITE,test_case,1},
 {testhelper,execute_test,3},
 {timer,tc,3},
 {testhelper,run_test2,4},
 {testhelper,'-run_test/4-fun-0-',6},
 {proc_lib,init_p,3}]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 
====================End of Test test_case========================
Running simple_test_SUITE:end_per_suite (runtime version)
****************************************************
simple_test_SUITE:test_case                                                                         failed (489us) Line 33 []
0 tests passed
1 tests failed
ok 

This time, testhelper tells us that test_case failed again, but on line 33 of the code. Much better.

Configuration Cases

Testhelper respects configuration cases. It treats each configuration case the same way Test Server does and passes state to the nested cases.

Configuration cases are basically a mechanism that helps with setup and the teardown of tests. They have the ability of changing the configuration passed to the tests embedded inside them.

Here is an example of a configuration case with a nested test case:

configuration_start(doc) ->
  ["Start of configuration case"];
configuration_start(suite) ->
  [];
configuration_start(Config) when is_list(Config) ->
  io:format("Configuration test case starts...\n"),
  Config.

test_case2(doc) -> 
  ["A 2nd Test Case"];

test_case2(suite) -> 
  [];

test_case2(Config) when is_list(Config) ->
  io:format("This is test 2!\n"),
  ok.

configuration_stop(doc) ->
  ["Start of configuration case"];
configuration_stop(suite) ->
  [];
configuration_stop(Config) when is_list(Config) ->
  io:format("Configuration test case ends!\n"),
  Config.

We also had to change the all(...) test case as follows:

all(doc) -> 
  ["This test links to all tests in this suite"]; 

all(suite) -> 
  [test_case,
   {conf,
    configuration_start,[test_case_2],
    configuration_stop}].

Note how we embedded our new test_case_2 inside a configuration case.

Now compile and run it:

(gdb_rdbms@wyemac)18> testhelper:run(simple_test_SUITE,all).
****************************************************
Running simple_test_SUITE:init_per_suite (runtime version) Case all 
====================Start of Test test_case======================
This is test 1!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Testhelper catch error {badmatch,the_new_atom}
[{simple_test_SUITE,test_case,1},
 {testhelper,execute_test,3},
 {timer,tc,3},
 {testhelper,run_test2,4},
 {testhelper,'-run_test/4-fun-0-',6},
 {proc_lib,init_p,3}]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================End of Test test_case========================
Starting Conf Case configuration_start...
====================Start of Test configuration_start======================
Configuration test case starts...
====================End of Test configuration_start========================
====================Start of Test test_case_2======================
This is test 2!
====================End of Test test_case_2========================
====================Start of Test configuration_stop======================
Configuration test case ends!
====================End of Test configuration_stop========================
Running simple_test_SUITE:end_per_suite (runtime version)
****************************************************
simple_test_SUITE:test_case                                                                         failed (501us) Line 37 []
2 tests passed
1 tests failed
ok

How it works

Testhelper runs each test in a separate process. This minimises the chances of test interfering with each other.

It doesn't print the test names that passed, as you can imaging printing out 40 tests that passed and one that failed you might just not be able to spot the broken one. On of course, our tests always works ;).

Testhelper skips any tests cases inside a configuration case if the start case failed, but will continue running the other tests if one nested one fails.

Download

testhelper.zip