Erlang Central

SideEffectsInGeneration

Revision as of 11:28, 30 December 2008 by TribbleFaith467 (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)


Authors

John Hughes posted by Thomas Arts

http://www.quviq.com/

Side-effects and Generation

Suppose you are testing a server with, among others, an Add and a Subtract operation, that respectively create an object on the server, and delete it again. Perhaps each operation has a wide variety of parameters, and you would like to test an Add-Subtract sequence before moving on to use eqc_statem. You write generators for the operation parameters, and away you go…

prop_add_sub() ->
   ?FORALL({AddParms,SubParms},{add_parameters(), sub_parameters()},
           begin 
             AddResult = add(AddParms),
             SubResult = subtract(SubParms),
             check_results(AddResult,SubResult)
           end).

But now suppose that Add returns the identifier of the newly added object, and that this identifier needs to passed in the parameters of Subtract in order to delete the right object. Of course, we need to pass AddResult to sub_parameters, so that it can be included in the generated parameters to subtract at the right point, but in the property above, AddResult is not available until after sub_parameters() has been called!

There is a simple and obvious solution… just rewrite the property with two ?FORALLs, and call add in between them.

prop_add_sub() ->
   ?FORALL(AddParms,add_parameters(),
           begin 
             AddResult = add(AddParms),
             ?FORALL(SubParms,sub_parameters(AddResult),
                     begin 
                       SubResult = subtract(SubParms),
                       check_results(AddResult,SubResult)
                     end)
           end).

Simple and obvious… but completely, dangerously wrong!

This property breaks a cardinal QuickCheck rule: it mixes side-effects with generation—before we finish generating the test case, we are already performing side-effects by calling Add. Why is this so dangerous? Because it makes generated tests unrepeatable. Remember that a test case is saved as the values of the variables bound in ?FORALLs. Presumably Add allocates a different object identifier each time it is called—so when this property is tested, the test case saved will contain the object identifier allocated by a particular call to Add. If that test is later repeated, then the repeated Add will generate a new object identifier, but the saved parameters to Subtract will still refer to the old one… leading, most likely, to an immediate failure. Repeatable test cases are critically important, both so that they can be traced and debugged, and because QuickCheck’s shrinking relies crucially on repeating variations on a test that just failed. If the repetitions fail for other, spurious, reasons, then shrinking will produce a spurious result.

So how can we avoid recording the object identifier that Add returns in the test case, but still use it when we perform the Subtract? The answer is to make the identifier symbolic. Let’s denote it by {var,object_id}, a QuickCheck symbolic variable. Now we can generate parameters for Subtract containing this symbolic variable instead of the actual object identifier—which means we can generate the entire test case before we perform any side-effects. All that remains is to replace the symbolic variable by the actual result of the Add, before the Subtract is performed. That can be done using eqc_gen:eval/2, which takes a property list and a term, and replaces symbolic variables in the term by values taken from the property list (version eqc-1.10 and later). Combining these ideas, we can rewrite the property correctly as shown.

prop_add_subtract() ->
   ?FORALL({AddParams,SubParams},
              {add_parameters(), sub_parameters({var,object_id})},
           begin 
             AddResult = add(AddParams),
             SubResult = subtract(eval([{object_id,AddResult}],SubParams)),
             check_results(AddResult,SubResult)
           end).

As you can see, the actual AddResult is passed to subtract by eval, thus letting us separate test case generation from execution in a clean way.

Of course, if you use eqc_statem to generate sequences of calls, rather than testing a specific sequence like this, then this is taken care of for you—but you can still see clearly from this example why symbolic variables, which eqc_statem also uses, are so important for getting repeatable test cases.

So remember the motto: never mix side-effects and test case generation.