As part of the toolkit for software development, writing (meaningful) automated tests is essential. Through unit, integration, and end-to-end tests, the proper functionality of our code is ensured, and we gain confidence that when implementing new requirements and refactoring existing code, we won’t break any existing functionality. Michael Feathers even defines legacy software as code that lacks tests.

legacy code is code without tests

– Michael Feathers

In addition to common approaches like example-based testing or snapshot testing, there are other methods that help us verify the correctness of our code. One of these approaches is property-based testing.

Property-based testing doesn’t check individual values for equality; rather, it aims to verify properties that our implementation must exhibit. For example, in mathematical addition, commutativity is a property that we can check without looking at specific numbers. For the addition of two numbers a and b, the commutative property always holds:

a + b = b + a

Property-based testing

Property testing or Property-based testing (PBT) is a testing technique that has become popular in functional programming, often associated with tools like QuickCheck in Haskell. It is well-suited for problem domains where verifying properties is much simpler than solving the problems themselves. However, it can also be valuable for testing more common scenarios.

In property-based testing, you define properties that the code under test must satisfy for a range of input data. Note that a property, in this context, isn’t the same as a C# property.

For instance, the property of commutativity in mathematical addition can be described as a function in C#:

Func<int, int, bool> commutativity = (int a, int b) => a + b == b + a;

The definition of Addition states that the commutative law holds true for all combinations of a and b. This means that our commutativity function must return true for all integer values.

When we implement our own Addition function, we can use this property to verify the correctness of our implementation.

Func<int, int, bool> commutativity = (int a, int b) 
    => MathOperations.Add(a, b) == MathOperations.Add(b, a);

Example-based testing

A naive approach to test the correctness of our Addition function is to call the function with a few example values and verify the return values, here using NUnit.

[TestCase(0, 0)]
[TestCase(1, 2)]
[TestCase(999, 9999)]
public void TestCommutativity_WithExamples(int a, int b)
{
    var left = MathOperations.Add(a, b);
    var right = MathOperations.Add(b, a);

    Assert.AreEqual(left, right);
}

Since we are not interested in the specific values of a and b, we can generate them randomly. This can be done for any number of combinations of a and b, in this case, it’s 1,000 iterations. By doing this, we have tested our Addition function with 1,000 randomly generated values instead of just three concrete values. This makes us more confident that our Addition function is working correctly.

[Test]
public void TestCommutativity_WithRandomValues()
{
    for (var i = 0; i < 1_000; i++)
    {
        var a = Random.Shared.Next(-999, 999);
        var b = Random.Shared.Next(-123, 321);

        var left = MathOperations.Add(a, b);
        var right = MathOperations.Add(b, a);

        Assert.AreEqual(left, right);
    }
}

However, this approach also comes with a few drawbacks: It’s clear that it requires more code, and our test case becomes more complex. Additionally, the generated values are chosen randomly and may not cover the entire range of possible input values.

A more significant issue is that if an error occurs, we won’t know which input values triggered the error since they are randomly generated in each iteration.

FsCheck

The library FsCheck written in F# provides a solution to this issue and can also be used in C#. The comprehensive documentation primarily uses F# syntax but also provides examples in C#.

It offers functions to simplify property testing. A test in FsCheck essentially consists of the following three parts:

> for all (x, y, ..)
> such as precondition(x, y, ...) holds 
> property(x, y, ...) is satisfied

This way, we can test the Commutativity property using FsCheck as follows:

[Test]
public void TestCommutativity()
{
    var property = (int x, int y)
        => MathOperations.Add(x, y) == MathOperations.Add(y, x);

    Prop.ForAll(
            Arb.From<int>(), // Parameter x
            Arb.From<int>(), // Parameter y
            property)
        .QuickCheckThrowOnFailure();
}

It becomes even easier when we use the extension FsCheck.NUnit (equivalent to FsCheck.Xunit for XUnit). Instead of annotating our test method with [Test] or [Fact], we annotate it with [Property] and can directly pass the properties as parameters. FsCheck automatically generates input values of the corresponding type.

[Property]
public bool TestCommutativity(int x, int y)
{
    return MathOperations.Add(x, y) == MathOperations.Add(y, x);
}

If we want to learn something about the distribution of the generated input values, we can classify them based on certain conditions. By calling Classify(), we get a value of type Property, so we need to adjust the return value of the test method accordingly.

[Property]
public Property TestCommutativity_WithClassification(int x, int y)
{
    var property = MathOperations.Add(x, y) == MathOperations.Add(y, x);

    return property
        .Classify(x == 0, "x == 0")
        .Classify(x > 0, "x > 0")
        .Classify(x < 0, "x < 0");
}

The corresponding output provides results similar to the following:

Ok, passed 100 tests.
50% x > 0.
46% x < 0.
4% x == 0.

If you want to have an output for each individual test case, the Collect() method can be helpful.

[Property]
public Property TestCommutativity_WithCollect(int x, int y)
{
    var property = MathOperations.Add(x, y) == MathOperations.Add(y, x);

    return property
        .Collect($"x: {x}, y: {y}");
}

The output provides a line for each combination along with its distribution.

Ok, passed 100 tests.
2% "x: 5, y: 9".
2% "x: 0, y: 0".
2% "x: -16, y: -6".
1% "x: 9, y: 32".
1% "x: 9, y: -26".
...

Conditions

If not all values of the data type of an input parameter are valid for a specific test case, we can use When() to define a precondition. To prevent dividing by 0 when testing Division, we exclude that case using this approach.

[Property]
public Property TestDivide(int x, int y)
{
    var property = () => MathOperations.Divide(x * y, y) == x;
    return property.When(y != 0);
}

Important to note: The definition of the property is executed as a method to allow for lazy evaluation by FsCheck.

Generator, Shrinker, Arbitrary

To generate suitable test data, FsCheck uses Generators, Shrinker, and Arbitraries. The Gen class provides the foundation for generating values with its three functions: Choose(), Constant(), and OneOf(). Values can be transformed and filtered using the Select() and Where() methods.

// Generate a list of numbers between 0 and 100 that are divisible by 2
var generator = Gen.Choose(0, 100)
    .Where(i => i % 2 == 0);

// Randomly choose one of the two values, "yes" or "no"
var generator = Gen.OneOf(
        Gen.Constant(true),
        Gen.Constant(false))
    .Select(b => b ? "yes" : "no");

The Sample() method of a generator provides a list of concrete values. The first parameter, size, has different effects depending on the generator, such as determining the range of values. In the case of Gen.Choose(), it has no effect. The second parameter, numberOfSamples, determines the number of elements in the generated list.

var sample = Gen.Choose(0, 100).Sample(0, 10);
// 22, 6, 16, 8, 27, 22, 14, 49, 42, 99

Shrinkers are used to simplify the process of finding errors. They attempt to find the smallest value for a failed test run, at which point the property being tested no longer holds true. In the following test, if the value is greater than or equal to 20, the test fails. By using shrinking, FsCheck can identify that the first value that causes the test to fail is 20.

[Property]
public bool Test_Shrink(PositiveInt value)
{
    return value.Item < 20;
}

Output:

Falsifiable, after 25 tests (2 shrinks) (StdGen (195734675,297194399)):
Original:
PositiveInt 24
Shrunk:
PositiveInt 20

Arbitraries combine Generators and Shrinkers and serve as inputs for property tests. The Arb class provides a set of default implementations for various data types. It can also be used to generate Generators for complex types.

var arbitrary1 = Arb.Default.PositiveInt();
var arbitrary2 = Arb.Default.DateTime();
var arbitrary3 = Arb.Default.IPv4Address();

var arbitrary4 = Arb.From<int>()
    .Filter(i => i % 2 == 0);

// --- 

Gen<Point> generator = Arb.Generate<Point>();

// --- 

Gen<Point> generator = from x in Arb.Default.PositiveInt().Generator
                       from y in Arb.Default.NegativeInt().Generator
                       select new Point(x.Item, y.Item);

The Property attribute’s properties can be used to control the distribution of values.

[Property(StartSize = 0, EndSize = 10)]
public Property Test_Distribution(int value)
{
    var property = () => true;

    return property.Collect($"value: {value}");
}

The values in the middle of the value range (here 0, since both positive and negative numbers are generated) are generated more frequently than the values at the edges. The specific values for StartSize and EndSize depend on the generator.

Ok, passed 100 tests.
18% "value: 0".
13% "value: 1".
9% "value: 2".
8% "value: -2".
7% "value: -4".
7% "value: -1".
5% "value: 5".
5% "value: 3".
5% "value: -8".
5% "value: -6".
4% "value: 4".
3% "value: 7".
2% "value: 9".
2% "value: 8".
2% "value: -7".
1% "value: 6".
1% "value: -9".
1% "value: -5".
1% "value: -3".
1% "value: -10".

Arguments exhausted after X tests

FsCheck performs 100 test runs by default. The MaxTest property allows you to control the number of runs. If you restrict the valid options too much, it’s possible that FsCheck won’t be able to generate any values and will throw an exception. This helps prevent excessive runtime for individual tests.

[Property(MaxTest = 100)]
public Property Test_Exhausted(int value)
{
    var property = () => true;

    return property
        .When(value % 15 == 0);
}

The test fails with a message:

Exhausted: Arguments exhausted after 61 tests.

Now you have the option to reduce the number of test runs. However, you should also consider whether you need to narrow down the valid range of values too much. In the example, using an int value is not a well-chosen data type.

Addition Example

To complete the example of addition, let’s now test the remaining properties of associativity, identity, and inverse.

[Property]
public bool TestCommutativity(int x, int y)
{
    // x + y == y + x
    return MathOperations.Add(x, y) == MathOperations.Add(y, x);
}

[Property]
public bool TestAssociativity(int x, int y, int z)
{
    // x + (y + z) == (x + y) + z
    return MathOperations.Add(x, MathOperations.Add(y, z)) ==
           MathOperations.Add(MathOperations.Add(x, y), z);
}

[Property]
public bool TestAdditiveIdentity(int x)
{
    // x + 0 == x
    return MathOperations.Add(x, 0) == x;
}

[Property]
public bool TestAdditiveInverse(int x)
{
    // x + (-x) == 0
    return MathOperations.Add(x, -x) == 0;
}

FizzBuzz Kata

For the well-known FizzBuzz kata, the objective is to generate one of the following strings for a positive number:

  • “Fizz” if the number is divisible by 3
  • “Buzz” if the number is divisible by 5
  • “FizzBuzz” if the number is divisible by both 3 and 5
  • the number itself, otherwise

A property-based test to verify a potential solution could look like this:

[Property]
public Property Should_Get_Fizz(PositiveInt n)
{
    var property = () => FizzBuzzer.GetFizzBuzz(n.Item).Equals("Fizz");

    return property.When(n.Item % 3 == 0 && n.Item % 5 != 0);
}

[Property]
public Property Should_Get_Buzz(PositiveInt n)
{
    var property = () => FizzBuzzer.GetFizzBuzz(n.Item).Equals("Buzz");

    return property.When(n.Item % 3 != 0 && n.Item % 5 == 0);
}

[Property(MaxTest = 10)]
public Property Should_Get_FizzBuzz(PositiveInt n)
{
    var property = () => FizzBuzzer.GetFizzBuzz(n.Item).Equals("FizzBuzz");

    return property.When(n.Item % 3 == 0 && n.Item % 5 == 0);
}

[Property]
public Property Should_Get_Number(PositiveInt n)
{
    var property = () => FizzBuzzer.GetFizzBuzz(n.Item).Equals(n.Item.ToString());

    return property.When(n.Item % 3 != 0 && n.Item % 5 != 0);
}

A small example project with the demonstrated source code can be found on GitHub.