06 February 2014

Acceptance Testing with FitNesse: Multiplicities and Comparisons

FitNesse is one of the most popular tools for unit testing since it is designed with a Wiki-style interface that makes it very easy to set up individual tests. Michael Sorens' sixth article in his series delves into the nuances of multiple inputs vs. multiple outputs, multiple rows vs. multiple columns, as well as things that can trip you up when attempting to validate a value.

Contents

  • Using FitNesse, Part 6
  • Dealing with Multiplicities
  • Comparisons
  • More to Come…
  • FitNesse is a wiki-based framework for writing acceptance tests for software systems. If you are not familiar with FitNesse, Part 1 of this series walks through a complete .NET example from writing the test in your browser to writing the C# code-behind. While FitNesse provides a rather nifty and user-friendly way to write acceptance tests in general, in practice there are plenty of quirks and glitches to watch out for. This and the subsequent parts of this series provide “tips from the trenches”, i.e. an accumulation of tips collected from intensive use of FitNesse on a daily basis to alleviate or avoid many of those pain points.

    Here is your roadmap to the series, showing where you are right now:

    Part 1: FitNesse Introduction and Walkthrough

    Part 2: Documentation and Infrastructure

    Part 3: Naming and Layout

    Part 4: Debugging, Control Flow, and Tracing

    Part 5: Symbols, Variables, and Code-Behind Style

    Part 6: Multiplicities and Comparisons

    Part 7: Database Fixtures, Project Overview

    Most sections in this article have references with actual hyperlinks to the FitNesse, fitSharp, or DbFit reference material. Some also have references to the sample test suite accompanying this series of articles, e.g. CleanCode.ConceptNotes.LayoutShowingEmbeddedNewlines. That path refers to a page on your FitNesse server. Thus if you are running on port 8080 on your local machine, the full URL to visit that page would be:

    http://localhost:8080/CleanCode.ConceptNotes.LayoutShowingEmbeddedNewlines

    Dealing with Multiplicities

    Combine Multiple Method Calls in One Test Table

    Do not use separate test tables for similar calls to the same method (i.e. calls that have the same signature). Compact your test vertically by combining calls to the same method in a single test table. This also adds additional structure to your tests by making it obvious that multiple calls to a method are, in fact, calling the same method.

    With Separate Tables

    Combined into a Single Table

    1936-imgF5.jpg


    !|Create Collection|
    |Add Value |Size?|
    |black,blue |2 |

    !|Create Collection|
    |Add Value |Size?|
    |green |1 |


    !|Create Collection|
    |Add Values |Size?|
    |black,blue |2 |
    |green |1 |

    When you have multiple calls to a single method, each is completely independent of the others and the order of execution is deterministic (i.e. the operations are performed sequentially in the order you specify). In the example shown, the Concat method simply concatenates the two values given with the specified separator. The output of the first call-stored in the FileName symbol-is used in both the second and third calls (highlighted in red to make this more obvious).

    1936-imgF5.jpg


    !|Concat |
    |Separator|Value1 |Value2 |Result? |
    |_ |<<ProgramNumber|${SEED}_badfile.xls|>>FileName |
    |\ |<<PaymentFolder|<<FileName |>>TestFile |
    |% |out of range |<<FileName |>>ErrMessage |

    Combine Multiple Database Queries in One Test Table

    A similar concept applies to combining multiple database query results. Rather than invoke the same query with a different parameter, make use of SQL language constructs. In this example use IN instead of a single EQUALS comparison in the WHERE clause. (Remember that within queries, @name is the notation to reference a symbol.)

    With Separate Tables

    Combined into a Single Table

    1936-imgF5.jpg


    !|Query|select SUM ...WHERE id="B" style="mso-bidi-font-weight: normal">@Name1|
    |Sum |Count |
    |${SumOfDetails1}|${CountOfDetails1} |

    !|Query|select SUM ...WHERE id="B" style="mso-bidi-font-weight: normal">@Name2|
    |Sum |Count |
    |${SumOfDetails2}|${CountOfDetails2} |

    !|Query|select SUM ...WHERE id="B" style="mso-bidi-font-weight: normal">@Name3|
    |Sum |Count |
    |${SumOfDetails3}|${CountOfDetails3} |


    !|Query|select SUM ...!-
    -!WHERE name in (@Name1,@Name2,@Name3)|
    |Sum |Count |
    |${SumOfDetails1} |${CountOfDetails1} |
    |${SumOfDetails2} |${CountOfDetails2} |
    |${SumOfDetails3} |${CountOfDetails3} |

    And, just as in the last section with multiple method calls, with multiple query calls (where you have some one or more expected fields to identify a row) you can order the rows how you like and then use results from earlier rows in later rows. Here the query stores the BatchNum corresponding to an Id of 1 and verifies the same BatchNum applies to an Id of 2.

    1936-imgF5.jpg


    !|query|select Id, BatchNum from ...|
    |Id |BatchNum? |
    |1 |>>Bat1 |
    |2 |<<Bat1 |

    Work With Multiple Outputs

    All the examples thus far have dealt with test tables that take one or more inputs but produce exactly one output. Part of the reason for that is that it is natural to imagine an equivalence relationship between a test table and a method, and a method returns just one output. That is fine for many cases. But you are not limited to that mental model. Rather, consider that you are communicating with a class rather a method, and you have much more flexibility.

    For each set of inputs, then, you can get not just one output but one row of output, though this output “row” is adjacent to-rather than under-the input “row”. Consider a simple Echo fixture class, shown below. I have contrived it for purposes of discussion to return up to two outputs, Result and ResultTimesTwo, for a given input. This is done by deriving from the ColumnFixture class, likely the most common pattern to use in FitNesse.

    Here is an example test table that does two separate operations. It first provides an input of “hello” and gets the results “hello” and “hello,hello”. The second invocation does the same for the string “world”.

    1936-imgF5.jpg


    !|Echo |
    |Value|Result?|Result Times Two?|
    |hello|hello |hello,hello |
    |world|world |world,world |

    1936-imgF8.jpg

    Echo

    Value

    Result?

    Result Times Two?

    hello

    hello

    hello,hello

    world

    world

    world,world

    For each input, you need either a public field (as shown here) or a publicly settable property. For each output, you need either a public field, a publicly readable property, or a public method.

    Here is one more example with one input and four outputs, which you may remember from Part 1 of this series (where you can also find the code-behind for the class):

    1936-imgF5.jpg


    !|Top Word |
    |Input |Word?|Occurrences?|Different Words?|Total Words?|
    |${InputText}|black|3 |4 |7 |

    Reference: Column Fixture

    Work With Multiple Output Rows

    The commonly used ColumnFixture class, as shown previously, allows you to get multiple return values (i.e. different columns) but returns only a single row per set of inputs. RowFixture, on the other hand, may be used to return any number of associated rows. In this example, the InvoiceDetailsFixture class inherits from RowFixture rather than ColumnFixture, taking no inputs and returning a series of rows each containing two outputs.

    1936-imgF5.jpg


    !|Invoice Details |
    |Invoice Number|Client Name |
    |A123 |>>Smith |
    |B456 |Wilma Jones |
    |C923 |<<Smith |
    |D111 |Casper |

    1936-imgF8.jpg

    Invoice Details

    Invoice Number

    Client Name

    A123

    >>Smith Fred Smith

    B456

    Wilma Jones

    C923

    << Smith Fred Smith

    D111 missing

    Casper

    F987 surplus

    George Bailey

    When executed this test validates all the rows specified; here the D111 row was not returned so is marked as missing, while one additional row (F987) was not expected, so is marked as surplus. Also note the use of symbols; one is collected from the first row and used to verify the third row. That is, just like a ColumnFixture, RowFixture results are ordered and you can use values from earlier rows in later rows.

    Here is the code-behind for this fixture class:

    If you compare the test to the fixture you can see how the fixture returns a series of Invoice objects to the test page. This is done quite simply by overriding the GetTargetClass method to identify the type of the return object, and by overriding the Query method to do whatever you want to generate a list of those objects. Here, the fixture is simply returning a hardcoded list; in practice you will return data generated either in code or retrieved from a database.

    Reference: Row Fixture, Fixture Arguments (FitNesse), Fixture Arguments (fitSharp)

    Comparisons

    Validate Approximate Values

    Up until now, you have seen test assertions done by filling in a cell with a value that must be matched exactly by the returned data; otherwise, that cell is marked as failing. FitSharp provides a number of cell operators that give you more expressiveness than just an equality check. For example, once you load any of these FitSharp string operators you can do partial string matches as shown. (Note that these cell operators are a replacement for the old cell handlers.)

    1936-imgF5.jpg


    !*> setup
    !|configuration setup |
    |service |
    |add operator|CompareStartsWith|
    |add operator|CompareEndsWith |
    |add operator|CompareSubstring |
    *!

    !|Echo |
    |Value |Result?|
    |abcdef|abc.. |
    |abcdef|..def |
    |abcdef|..def..|

    1936-imgF8.jpg

    setup

    Echo

    Value

    Result?

    abcdef

    abc..

    abcdef

    ..def

    abcdef

    ..def..

    1936-imgFC.gif

    setup

    Echo

    Value

    Result?

    abcdef

    abc..

    abcdef

    ..def

    abcdef

    ..def..

    Other operators allow comparing strings with options to ignore white space and/or case (CompareString), asserting that an integer lies within a given range (CompareIntegralRange) or match a string to a regular expression (CompareRegEx).

    References: fitSharp Cell Operators, Cell Operators for Simpler Comparisons, CleanCode.GeneralUtilityFixtureNotes.EchoFixture

    Be Aware of the Precision of Your Database Values

    Take care with specifying precision when you retrieve and compare certain data types from a database. Besides the obvious consideration of floating point numbers there are other key types to consider as well, such as dates and times. This statement…

    …retrieves the time down to the millisecond-probably not what you want! Rather convert it to a precision relevant to the task at hand. If you only want to validate the date and ignore the time altogether, use something like this (highlighted in red for emphasis):

    1936-imgF5.jpg


    !|Query|!-SELECT RecordID, WidgetID, Price, Active,
    CONVERT(char(10),DateValidFrom,101) [DateValidFrom],
    DateValidFrom as [DateValidFrom_Original]
    -!FROM dbo.${MyTable}!-
    -!WHERE Price > @TargetPrice|
    |RecordID |WidgetID|Price|Active|DateValidFrom|DateValidFrom_Original|
    |<<record1|25 |1010 | |01/01/2013 | |

    1936-imgF8.jpg

    Query


    SELECT RecordID, WidgetID, Price, Active,
    CONVERT(char(10),DateValidFrom,101) [DateValidFrom],
    DateValidFrom as [DateValidFrom_Original]
    FROM dbo.MyWidgets_tmp
    WHERE Price > @TargetPrice

    RecordID

    WidgetID

    Price

    Active

    DateValidFrom

    DateValidFrom_Original

    <<record1

    25

    1010

    01/01/2013

    <<record2

    12

    1011

    05/01/2013

    1936-imgFC.gif

    Query


    SELECT RecordID, WidgetID, Price, Active,
    CONVERT(char(10),DateValidFrom,101) [DateValidFrom],
    DateValidFrom as [DateValidFrom_Original]
    FROM dbo.MyWidgets_tmp
    WHERE Price > @TargetPrice

    RecordID

    WidgetID

    Price

    Active

    DateValidFrom

    DateValidFrom_Original

    <<record1 1

    25

    1010

    null

    01/01/2013

    01/01/2013 10:21:03

    <<record2 2

    12

    1011

    null

    05/01/2013

    05/01/2013 08:15:00

    I often include the complete, original value as well for my own benefit, not for validation purposes, as the example illustrates.

    References: TSQL Cast and Convert, CleanCode.DataBaseNotes.CrudOperations

    Watch Out For Whitespace

    Watch out for whitespace when you retrieve and compare strings from a database. Unless you know a particular field is well-behaved with respect to whitespace assume it is not-program defensively. Consider the example below. If the description is a value the user typed in, then the user may have typed “dry goods” or ” dry goods” or “dry goods” or “dry goods “. Only the first one would produce a positive match against invoice 101 on this test table:

    You might see the problem if the second or third variation had been typed, but you will likely not see it if the last one is typed (the one with trailing spaces). To obfuscate the problem even further, the test table itself ignores trailing spaces in what you enter. That is all of these lines are equivalent in a test table:

    They will match only “dry goods” in the database though so if the database actually contains “dry goods ” you will think FitNesse is broken because in essence it reports “dry goods” does not match “dry goods”! The solution is to use the RTRIM function in SQL:

    Know What Comprises Your Database Key

    In a non-database test table, the presence of a trailing question mark or trailing parentheses attached to a column name is what distinguishes inputs from outputs. In a database test table, the presence of a trailing question mark indicates a field that is not part of the record’s composite key; omitting the question mark includes it in the composite key. This partial key matching allows non-key values to be compared yet not used in identifying a row. Here is the distinction. In this first example, columns C and D are marked as non-key fields but they are validated because we put values in the table cells.

    1936-imgF5.jpg


    !|query|SELECT 1 ColA, 2 ColB, 3 ColC, 4 ColD!-
    -! UNION SELECT 21, 22, 23, 24!-
    -! UNION SELECT 31, 32, 33, 34|
    |ColA|ColB|ColC?|ColD?|
    |1 |2 |3 |4 |
    |21 |22 |21 |21 |
    |31 |32 |33 |34 |

    1936-imgFC.gif

    query


    SELECT 1 ColA, 2 ColB, 3 ColC, 4 ColD
    UNION SELECT 21, 22, 23, 24
    UNION SELECT 31, 32, 33, 34

    ColA

    ColB

    ColC?

    ColD?

    1

    2

    3

    4

    21

    22


    21 expected
    ---------------
    23 actual

    21 expected
    ---------------
    24 actual

    31

    32

    33

    34

    If you omit question marks from the non-key columns they cause rows to be rejected completely-at least this is the way it worked in a prior release (for .NET) and still does (for Java). I wanted to include this section precisely because of this current difference between Java and .NET implementations-something to be aware of.

    1936-imgF5.jpg


    !|query|SELECT 1 ColA, 2 ColB, 3 ColC, 4 ColD!-
    -! UNION SELECT 21, 22, 23, 24!-
    -! UNION SELECT 31, 32, 33, 34|
    |ColA|ColB|ColC|ColD|
    |1 |2 |3 |4 |
    |21 |22 |21 |21 |
    |31 |32 |33 |34 |

    1936-imgFC.gif

    query


    SELECT 1 ColA, 2 ColB, 3 ColC, 4 ColD
    UNION SELECT 21, 22, 23, 24
    UNION SELECT 31, 32, 33, 34

    ColA

    ColB

    ColC

    ColD

    1

    2

    3

    4

    21 missing

    22

    21

    21

    31

    32

    33

    34

    21 surplus

    22

    23

    24

    References: CleanCode.DataBaseNotes.SpecifyingFieldsToMatch

    More to Come…

    Part 7 provides details on database operations as well as an overview of the fixture library and sample test suite accompanying this series of articles.

Keep up to date with Simple-Talk

For more articles like this delivered fortnightly, sign up to the Simple-Talk newsletter

Downloads

This post has been viewed 9340 times – thanks for reading.

Tags: ,

  • Rate
    [Total: 3    Average: 5/5]
  • Share

Michael Sorens is passionate about software to be more productive, evidenced by his open source libraries in several languages (see his API bookshelf) as well as SqlDiffFramework (a DB comparison tool for heterogeneous systems including SQL Server, Oracle, and MySql). With degrees in computer science and engineering he has worked the gamut of companies from Fortune 500 firms to Silicon Valley startups over the last 25 years or so. Current passions include PowerShell, .NET, SQL, and XML technologies (see his full brand page). Spreading the seeds of good design wherever possible, he enjoys sharing knowledge via writing (see his full list of articles), teaching, and StackOverflow. Like what you have read? Connect with Michael on LinkedIn and Google +

View all articles by Michael Sorens