Erlang Central

Difference between revisions of "Testing with Test Helper"

From ErlangCentral Wiki

(Create Page)
 
m
(7 intermediate revisions by one user not shown)
Line 1: Line 1:
 
[[Category:Testing]]
 
[[Category:Testing]]
 
[[Category:HowTo]]
 
[[Category:HowTo]]
==Author==
+
==Authors==
 
[[User:Rvg|Rudolph van Graan]]
 
[[User:Rvg|Rudolph van Graan]]
  
 
==Article==
 
==Article==
 +
There are several test philosophies out there. We use a combination of [[XP|Extreme Programming]] and [[DSDM]]. We don't ever write code without writing a test first. In fact, we use the test to state what we want the code to do.
 
For this you can use the erlang test server. It's home page is http://www.erlang.org/project/test_server/index.html. It is quite an involved process getting it to work, as you can find on [http://www.erlang.org/project/test_server/README this page].
 
For this you can use the erlang test server. It's home page is http://www.erlang.org/project/test_server/index.html. It is quite an involved process getting it to work, as you can find on [http://www.erlang.org/project/test_server/README this page].
  
In order to speed up the testing process and to fit in with a philosophy of test as you go, we created [[File:testhelper.erl]]. It makes it easy to build test suites that are compatible with the [[Test Server]], but still allows you to run your tests in [[Emacs]].
+
In order to speed up the testing process and to fit in with a philosophy of test as you go, we created <tt>testhelper.erl</tt>. It makes it easy to build test suites that are compatible with the [[Test Server]], but still allows you to run your tests in [[Emacs]]. It doesn't support all the features of the Test Server, but does make it easier to test and develop on the go.
  
 
==Writing a test suite==
 
==Writing a test suite==
Line 13: Line 14:
  
 
Here is the quick recipe:
 
Here is the quick recipe:
 +
# Generate the skeleton in Emacs
 +
# Create a number of tests
 +
# Run the test suite using testhelper
  
1. In [[Emacs]] select Erlang|Skeletons|Erlang test suite TS Frontend:
+
===Generate the skeleton===
 +
1. In Emacs, create a file named <tt>simple_test_SUITE.erl</tt><br>
 +
2. In [[Emacs]] select Erlang|Skeletons|Erlang test suite TS Frontend:
  
 
[[Image:testhelper_step_1.png]]
 
[[Image:testhelper_step_1.png]]
  
 +
This will fill in your current module with a basic skeleton. If we remove all the comments, the test should look like this:
  
==Running all the tests in the suite==
+
-module(simple_test_SUITE).
In order to run our new test suite, we
+
 +
-compile(export_all).
 +
-include("test_server/include/test_server.hrl").
 +
 +
init_per_suite(Config) ->
 +
  Config.
 +
 +
end_per_suite(_Config) ->
 +
  ok.
 +
 +
init_per_testcase(_TestCase, Config) ->
 +
  Config.
 +
 +
end_per_testcase(_TestCase, _Config) ->
 +
  ok.
 +
 +
all(doc) ->
 +
  ["This test links to all tests in this suite"];
 +
 +
all(suite) ->
 +
  [test_case].
 +
 +
test_case(doc) ->
 +
  ["Describe the main purpose of test case"];
 +
 +
test_case(suite) ->
 +
  [];
 +
 +
test_case(Config) when is_list(Config) ->
 +
  ok.
  
 +
3. Next, compile the suite:
  
 +
(gdb_rdbms@wyemac)1> c("simple_test_SUITE",[debug_info]).
 +
{ok,simple_test_SUITE}
 +
 +
 +
===Running all the tests in the new suite===
 +
In order to run our new test suite, we simply call testhelper like this:
 +
 +
(gdb_rdbms@wyemac)2> testhelper:run(simple_test_SUITE,all).
 +
****************************************************
 +
Running simple_test_SUITE:init_per_suite (runtime version) Case all
 +
DataDir = "/Users/rvg/svn/modules/common/test/simple_test_SUITE_data/"
 +
Config = [{data_dir,"./simple_test_SUITE_data/"},
 +
          {priv_dir,"./simple_test_SUITE_priv/"}]
 +
 +
 +
====================Start of Test test_case======================
 +
====================End of Test test_case========================
 +
Running simple_test_SUITE:end_per_suite (runtime version)
 +
****************************************************
 +
1 tests passed
 +
0 tests failed
 +
ok
 +
 +
At this stage you'll notice that testhelper indicated 1 tests passed, 0 failed. The test that passed was our <tt>test_case(...)</tt> created above. Of course it doesn't do anything yet.
 +
 +
Let's make it fail. Change test_case(...) as follows:
 +
 +
test_case(doc) ->
 +
  ["Describe the main purpose of test case"];
 +
 +
test_case(suite) ->
 +
  [];
 +
 +
test_case(Config) when is_list(Config) ->
 +
  My1stAtom = the_new_atom,
 +
  My2ndAtom = the_other_atom,
 +
  My2ndAtom = My1stAtom,
 +
  ok.
 +
 +
The test simply creates two atoms, one named <tt>My1stAtom</tt>, the other <tt>My2ndAtom</tt>. In the third line, we make an assumption that these two atoms are the same by trying to match the two variables. (See my article [[Hardcoding Expectations]])
 +
 +
Lets test this. Run testhelper again:
 +
 +
(gdb_rdbms@wyemac)4> testhelper:run(simple_test_SUITE,all).
 +
****************************************************
 +
Running simple_test_SUITE:init_per_suite (runtime version) Case all
 +
====================Start of Test test_case======================
 +
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 +
Testhelper catch error {badmatch,the_new_atom}
 +
[{simple_test_SUITE,test_case,1},
 +
  {timer,tc,3},
 +
  {testhelper,run_test2,4},
 +
  {testhelper,'-run_test/4-fun-0-',6},
 +
  {proc_lib,init_p,3}]
 +
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 +
====================End of Test test_case========================
 +
Running simple_test_SUITE:end_per_suite (runtime version)
 +
****************************************************
 +
simple_test_SUITE:test_case                                                                        failed (84ms) (No Line) []
 +
0 tests passed
 +
1 tests failed
 +
ok
 +
 +
This time, one test failed. You can see in the debug output that a [[Bad Match|bad match]] error occurred.
 +
The reason for this is that we tried to match two [[Atom]]s and they are just not created equal.
 +
 +
Unfortunately, we have to figure out where the error occurred (The third line of the third function clause), but in larger test suites, it becomes difficult to trace errors.
 +
Let us fix this. Change the code and recompile:
 +
 +
test_case(Config) when is_list(Config) ->
 +
  ?line My1stAtom = the_new_atom,
 +
  ?line My2ndAtom = the_other_atom,
 +
  ?line My2ndAtom = My1stAtom,
 +
  ok.
 +
 +
The [[Line Macro|?line macro]] is a mechanism that helps you find the error. Run testhelper again:
 +
 +
(gdb_rdbms@wyemac)6> testhelper:run(simple_test_SUITE,all).
 +
****************************************************
 +
====================Start of Test test_case======================
 +
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 +
Testhelper catch error {badmatch,the_new_atom}
 +
[{simple_test_SUITE,test_case,1},
 +
  {testhelper,execute_test,3},
 +
  {timer,tc,3},
 +
  {testhelper,run_test2,4},
 +
  {testhelper,'-run_test/4-fun-0-',6},
 +
  {proc_lib,init_p,3}]
 +
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 +
====================End of Test test_case========================
 +
Running simple_test_SUITE:end_per_suite (runtime version)
 +
****************************************************
 +
simple_test_SUITE:test_case                                                                        failed (489us) Line 33 []
 +
0 tests passed
 +
1 tests failed
 +
ok
 +
 +
This time, testhelper tells us that test_case failed again, but on line 33 of the code. Much better.
 +
 +
===Configuration Cases===
 +
Testhelper respects [http://www.erlang.org/project/test_server/html/test_spec_chapter.html#3.4 configuration cases]. It treats each configuration case the same way [[Test Server]] does and passes state to the nested cases.
 +
 +
Configuration cases are basically a mechanism that helps with setup and the teardown of tests. They have the ability of changing the configuration passed to the tests embedded inside them.
 +
 +
Here is an example of a configuration case with a nested test case:
 +
 +
configuration_start(doc) ->
 +
  ["Start of configuration case"];
 +
configuration_start(suite) ->
 +
  [];
 +
configuration_start(Config) when is_list(Config) ->
 +
  io:format("Configuration test case starts...\n"),
 +
  Config.
 +
 +
test_case2(doc) ->
 +
  ["A 2nd Test Case"];
 +
 +
test_case2(suite) ->
 +
  [];
 +
 +
test_case2(Config) when is_list(Config) ->
 +
  io:format("This is test 2!\n"),
 +
  ok.
 +
 +
configuration_stop(doc) ->
 +
  ["Start of configuration case"];
 +
configuration_stop(suite) ->
 +
  [];
 +
configuration_stop(Config) when is_list(Config) ->
 +
  io:format("Configuration test case ends!\n"),
 +
  Config.
 +
 +
We also had to change the all(...) test case as follows:
 +
 +
all(doc) ->
 +
  ["This test links to all tests in this suite"];
 +
 +
all(suite) ->
 +
  [test_case,
 +
    {conf,
 +
    configuration_start,[test_case_2],
 +
    configuration_stop}].
 +
 +
Note how we embedded our new test_case_2 inside a configuration case.
 +
 +
Now compile and run it:
 +
 +
(gdb_rdbms@wyemac)18> testhelper:run(simple_test_SUITE,all).
 +
****************************************************
 +
Running simple_test_SUITE:init_per_suite (runtime version) Case all
 +
====================Start of Test test_case======================
 +
This is test 1!
 +
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 +
Testhelper catch error {badmatch,the_new_atom}
 +
[{simple_test_SUITE,test_case,1},
 +
  {testhelper,execute_test,3},
 +
  {timer,tc,3},
 +
  {testhelper,run_test2,4},
 +
  {testhelper,'-run_test/4-fun-0-',6},
 +
  {proc_lib,init_p,3}]
 +
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 +
====================End of Test test_case========================
 +
Starting Conf Case configuration_start...
 +
====================Start of Test configuration_start======================
 +
Configuration test case starts...
 +
====================End of Test configuration_start========================
 +
====================Start of Test test_case_2======================
 +
This is test 2!
 +
====================End of Test test_case_2========================
 +
====================Start of Test configuration_stop======================
 +
Configuration test case ends!
 +
====================End of Test configuration_stop========================
 +
Running simple_test_SUITE:end_per_suite (runtime version)
 +
****************************************************
 +
simple_test_SUITE:test_case                                                                        failed (501us) Line 37 []
 +
2 tests passed
 +
1 tests failed
 +
ok
 +
 +
==How it works==
 +
Testhelper runs each test in a separate process. This minimises the chances of test interfering with each other.
 +
 +
It doesn't print the test names that passed, as you can imaging printing out 40 tests that passed and one that failed you might just not be able to spot the broken one. On of course, our tests always works ;).
 +
 +
Testhelper skips any tests cases inside a configuration case if the start case failed, but will continue running the other tests if one nested one fails.
 +
 +
==See Also==
 +
* My [[Debugging with Debug Helper|article on debugging]] tests
 +
* [[Test Driven Development]] in Erlang
 
==Download==
 
==Download==
 
[http://www.patternmatched.com/download/testhelper.zip testhelper.zip]
 
[http://www.patternmatched.com/download/testhelper.zip testhelper.zip]

Revision as of 07:29, 3 September 2006

Contents

Authors

Rudolph van Graan

Article

There are several test philosophies out there. We use a combination of Extreme Programming and DSDM. We don't ever write code without writing a test first. In fact, we use the test to state what we want the code to do. For this you can use the erlang test server. It's home page is http://www.erlang.org/project/test_server/index.html. It is quite an involved process getting it to work, as you can find on this page.

In order to speed up the testing process and to fit in with a philosophy of test as you go, we created testhelper.erl. It makes it easy to build test suites that are compatible with the Test Server, but still allows you to run your tests in Emacs. It doesn't support all the features of the Test Server, but does make it easier to test and develop on the go.

Writing a test suite

The basic steps are exactly the same as for Test Server. For details, look at Writing Test Suites

Here is the quick recipe:

  1. Generate the skeleton in Emacs
  2. Create a number of tests
  3. Run the test suite using testhelper

Generate the skeleton

1. In Emacs, create a file named simple_test_SUITE.erl
2. In Emacs select Erlang|Skeletons|Erlang test suite TS Frontend:

Testhelper step 1.png

This will fill in your current module with a basic skeleton. If we remove all the comments, the test should look like this:

-module(simple_test_SUITE).

-compile(export_all).
-include("test_server/include/test_server.hrl").

init_per_suite(Config) ->
  Config.

end_per_suite(_Config) ->
  ok.

init_per_testcase(_TestCase, Config) ->
  Config.

end_per_testcase(_TestCase, _Config) ->
  ok.

all(doc) -> 
  ["This test links to all tests in this suite"];

all(suite) -> 
  [test_case].

test_case(doc) -> 
  ["Describe the main purpose of test case"];

test_case(suite) -> 
  [];

test_case(Config) when is_list(Config) -> 
  ok.

3. Next, compile the suite:

(gdb_rdbms@wyemac)1> c("simple_test_SUITE",[debug_info]).
{ok,simple_test_SUITE}


Running all the tests in the new suite

In order to run our new test suite, we simply call testhelper like this:

(gdb_rdbms@wyemac)2> testhelper:run(simple_test_SUITE,all).
****************************************************
Running simple_test_SUITE:init_per_suite (runtime version) Case all 
DataDir = "/Users/rvg/svn/modules/common/test/simple_test_SUITE_data/" 
Config = [{data_dir,"./simple_test_SUITE_data/"},
          {priv_dir,"./simple_test_SUITE_priv/"}]


====================Start of Test test_case======================
====================End of Test test_case========================
Running simple_test_SUITE:end_per_suite (runtime version)
****************************************************
1 tests passed
0 tests failed
ok

At this stage you'll notice that testhelper indicated 1 tests passed, 0 failed. The test that passed was our test_case(...) created above. Of course it doesn't do anything yet.

Let's make it fail. Change test_case(...) as follows:

test_case(doc) -> 
  ["Describe the main purpose of test case"];

test_case(suite) -> 
  [];

test_case(Config) when is_list(Config) -> 
  My1stAtom = the_new_atom,
  My2ndAtom = the_other_atom,
  My2ndAtom = My1stAtom,
  ok.

The test simply creates two atoms, one named My1stAtom, the other My2ndAtom. In the third line, we make an assumption that these two atoms are the same by trying to match the two variables. (See my article Hardcoding Expectations)

Lets test this. Run testhelper again:

(gdb_rdbms@wyemac)4> testhelper:run(simple_test_SUITE,all).
****************************************************
Running simple_test_SUITE:init_per_suite (runtime version) Case all 
====================Start of Test test_case======================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Testhelper catch error {badmatch,the_new_atom}
[{simple_test_SUITE,test_case,1},
 {timer,tc,3},
 {testhelper,run_test2,4},
 {testhelper,'-run_test/4-fun-0-',6},
 {proc_lib,init_p,3}]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================End of Test test_case========================
Running simple_test_SUITE:end_per_suite (runtime version)
****************************************************
simple_test_SUITE:test_case                                                                         failed (84ms) (No Line) []
0 tests passed
1 tests failed
ok

This time, one test failed. You can see in the debug output that a bad match error occurred. The reason for this is that we tried to match two Atoms and they are just not created equal.

Unfortunately, we have to figure out where the error occurred (The third line of the third function clause), but in larger test suites, it becomes difficult to trace errors. Let us fix this. Change the code and recompile:

test_case(Config) when is_list(Config) -> 
  ?line My1stAtom = the_new_atom,
  ?line My2ndAtom = the_other_atom,
  ?line My2ndAtom = My1stAtom,
  ok.

The ?line macro is a mechanism that helps you find the error. Run testhelper again:

(gdb_rdbms@wyemac)6> testhelper:run(simple_test_SUITE,all).
****************************************************
====================Start of Test test_case======================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Testhelper catch error {badmatch,the_new_atom}
[{simple_test_SUITE,test_case,1},
 {testhelper,execute_test,3},
 {timer,tc,3},
 {testhelper,run_test2,4},
 {testhelper,'-run_test/4-fun-0-',6},
 {proc_lib,init_p,3}]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 
====================End of Test test_case========================
Running simple_test_SUITE:end_per_suite (runtime version)
****************************************************
simple_test_SUITE:test_case                                                                         failed (489us) Line 33 []
0 tests passed
1 tests failed
ok 

This time, testhelper tells us that test_case failed again, but on line 33 of the code. Much better.

Configuration Cases

Testhelper respects configuration cases. It treats each configuration case the same way Test Server does and passes state to the nested cases.

Configuration cases are basically a mechanism that helps with setup and the teardown of tests. They have the ability of changing the configuration passed to the tests embedded inside them.

Here is an example of a configuration case with a nested test case:

configuration_start(doc) ->
  ["Start of configuration case"];
configuration_start(suite) ->
  [];
configuration_start(Config) when is_list(Config) ->
  io:format("Configuration test case starts...\n"),
  Config.

test_case2(doc) -> 
  ["A 2nd Test Case"];

test_case2(suite) -> 
  [];

test_case2(Config) when is_list(Config) ->
  io:format("This is test 2!\n"),
  ok.

configuration_stop(doc) ->
  ["Start of configuration case"];
configuration_stop(suite) ->
  [];
configuration_stop(Config) when is_list(Config) ->
  io:format("Configuration test case ends!\n"),
  Config.

We also had to change the all(...) test case as follows:

all(doc) -> 
  ["This test links to all tests in this suite"]; 

all(suite) -> 
  [test_case,
   {conf,
    configuration_start,[test_case_2],
    configuration_stop}].

Note how we embedded our new test_case_2 inside a configuration case.

Now compile and run it:

(gdb_rdbms@wyemac)18> testhelper:run(simple_test_SUITE,all).
****************************************************
Running simple_test_SUITE:init_per_suite (runtime version) Case all 
====================Start of Test test_case======================
This is test 1!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Testhelper catch error {badmatch,the_new_atom}
[{simple_test_SUITE,test_case,1},
 {testhelper,execute_test,3},
 {timer,tc,3},
 {testhelper,run_test2,4},
 {testhelper,'-run_test/4-fun-0-',6},
 {proc_lib,init_p,3}]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================End of Test test_case========================
Starting Conf Case configuration_start...
====================Start of Test configuration_start======================
Configuration test case starts...
====================End of Test configuration_start========================
====================Start of Test test_case_2======================
This is test 2!
====================End of Test test_case_2========================
====================Start of Test configuration_stop======================
Configuration test case ends!
====================End of Test configuration_stop========================
Running simple_test_SUITE:end_per_suite (runtime version)
****************************************************
simple_test_SUITE:test_case                                                                         failed (501us) Line 37 []
2 tests passed
1 tests failed
ok

How it works

Testhelper runs each test in a separate process. This minimises the chances of test interfering with each other.

It doesn't print the test names that passed, as you can imaging printing out 40 tests that passed and one that failed you might just not be able to spot the broken one. On of course, our tests always works ;).

Testhelper skips any tests cases inside a configuration case if the start case failed, but will continue running the other tests if one nested one fails.

See Also

Download

testhelper.zip