7 Integration testing

This chapter covers

In chapter 6, we discussed unit testing in Rust. In this chapter, we’ll discuss how to use integration testing in Rust, and how it compares to unit testing. Both unit testing and integration testing are powerful strategies to improve the quality of your software. They are often used together with slightly different goals.

Integration testing can sometimes be a little more difficult because it may require more work to create harnesses and test cases, depending on the type of software being tested. It’s more common to find unit tests than integration tests, but Rust provides the basic tools you need to write effective integration tests without spending too much time on boilerplate and harnesses. We’ll also explore some libraries that can help turbocharge your integration testing without requiring much additional work.

7.1 Comparing integration and unit testing

Integration testing is the testing of individual modules or groups from their public interfaces. This is in contrast to unit testing, which is the testing of the smallest testable components within software, sometimes including private interfaces. Public interfaces are those which are exposed to external consumers of software, such as the public library interfaces or the CLI commands, in the case of a command line application.

In Rust, integration tests share very little with unit tests. Unlike unit tests, integration tests are located outside of the main source tree. Rust treats integration tests as a separate crate; thus, they only have access to the publicly exported functions and structures.

Let’s write a quick, generic implementation of the quicksort algorithm (https://en.wikipedia.org/wiki/Quicksort), which many of us know and love from our computer science studies, as shown in the following listing.

Listing 7.1 Quicksort implemented in Rust

pub fn quicksort<T: std::cmp::PartialOrd + Clone>(slice: &mut [T]) {   
    if slice.len() < 2 {
        return;
    }
    let (left, right) = partition(slice);
    quicksort(left);
    quicksort(right);
}
 
fn partition<T: std::cmp::PartialOrd + Clone>(
    slice: &mut [T]
) -> (&mut [T], &mut [T]) {                                            
    let pivot_value = slice[slice.len() - 1].clone();
    let mut pivot_index = 0;
    for i in 0..slice.len() {
        if slice[i] <= pivot_value {
            slice.swap(i, pivot_index);
            pivot_index += 1;
        }
    }
    if pivot_index < slice.len() - 1 {
        slice.swap(pivot_index, slice.len() - 1);
    }
 
    slice.split_at_mut(pivot_index - 1)
}

This is our public quicksort() function, denoted as such by the pub keyword.

Our private partition() function is not accessible outside of the local scope.

Integration tests are located within the tests directory, at the top level of the source tree. These tests are automatically discovered by Cargo. An example directory structure for a small library (in src/lib.rs) and a single integration test would look like this:

$ tree
.
├── Cargo.lock
├── Cargo.toml
├── src
│   └── lib.rs           
└── tests                
    └── quicksort.rs     
 
2 directories, 4 files

Contains the source code for our library

Integration tests are located within the tests directory.

quicksort.rs contains our integration tests.

Test functions are marked with the #[test] attribute, which will be run by Cargo automatically. You can either use the automatically provided main() function from libtest or supply your own, as with unit tests. Cargo handles these integration tests as separate crates. You can have multiple separate sets of crates by creating directories within the tests directory, each containing their own separate integration tests.

As with unit tests, we typically use the assertion macros (assert!() and assert_eq!()) to verify results. The integration is shown in practice in the following listing.

Listing 7.2 Code sample of the integration test for quicksort implementation

use quicksort::quicksort;
 
#[test]
fn test_quicksort() {
    let mut values = vec![12, 1, 5, 0, 6, 2];
    quicksort(&mut values);
    assert_eq!(values, vec![0, 1, 2, 5, 6, 12]);
 
    let mut values = vec![1, 13, 5, 10, 6, 2, 0];
    quicksort(&mut values);
    assert_eq!(values, vec![0, 1, 2, 5, 6, 10, 13]);
}

This looks a lot like unit tests, yes? The difference in an example like this is almost entirely semantics—in fact, this code sample includes unit tests, which look nearly the same in the following listing.

Listing 7.3 Code sample of unit tests for quicksort implementation

#[cfg(test)]
mod tests {
    use crate::{partition, quicksort};
 
    #[test]
    fn test_partition() {
        let mut values = vec![0, 1, 2, 3];
        assert_eq!(
            partition(&mut values),
            (vec![0, 1, 2].as_mut_slice(), vec![3].as_mut_slice())
        );
 
        let mut values = vec![0, 1, 2, 4, 3];
        assert_eq!(
            partition(&mut values),
            (vec![0, 1, 2].as_mut_slice(), vec![3, 4].as_mut_slice())
        );
    }
 
    #[test]
    fn test_quicksort() {
        let mut values = vec![1, 5, 0, 6, 2];
        quicksort(&mut values);
        assert_eq!(values, vec![0, 1, 2, 5, 6]);
 
        let mut values = vec![1, 5, 10, 6, 2, 0];
        quicksort(&mut values);
        assert_eq!(values, vec![0, 1, 2, 5, 6, 10]);
    }
}

The only real difference with this code is that in the unit tests, we’re also testing the partition() function, which is nonpublic. Is this a case where we shouldn’t bother writing integration tests? No. Why? Because we’re creating a library with a public interface, and we should test the library, as it’s intended to be used externally. Integration tests live outside the library (or application) we’re testing; thus, they only have visibility to public (and external) interfaces. This forces us to write tests the same way that downstream users of our library or application would use the software. Integration testing helps us make sure the public API works as intended from the perspective of external users.

7.2 Integration testing strategies

It wasn’t too long ago that test driven development (TDD) was all the rage. TDD is based on the idea that you write your tests before writing software. The theory of TDD is that writing tests first helps you build quality code faster. TDD seems to have fallen out of favor, but it does provide us with some insights, especially regarding integration testing.

One thing we can learn from TDD is that designing APIs is just as important as the testing itself. The ergonomics of your software matters, whether you’re building libraries; command line applications; or web, desktop, or mobile apps. The end user experience (UX) is surfaced when writing integration tests; these tests force you to think about how your software is used from the perspective of the person using it.

Integration testing and unit testing aren’t mutually exclusive; they should be used to complement each other where appropriate. Integration tests shouldn’t be written the same way as unit tests because we’re testing different things. When we think about writing integration tests, we need to consider more than just the correctness of an algorithm or the logic it implements.

We should think about integration tests not only as a way of verifying our code works but also as a way of testing the UX of our software. There are plenty of examples of good and bad software design, and the process of writing integration tests for your own software forces you to sample a taste of your design. It’s easy to get tunnel vision and lose sight of the big picture when writing software, and integration tests are, by definition, a holistic perspective of your code.

I’ve personally experienced this tunnel vision problem many times. For example, when writing the dryoc crate, I got a little carried away with some of the optional features, and it wasn’t until I tried to write integration tests that I realized I’d done a poor job of designing the interface at the time. I had to refactor my design substantially to make the library easier to use.

Regarding TDD, should you write integration tests before writing your library or application? This is not a practice that I follow, but I don’t believe it’s necessarily bad, so I recommend testing and determining whether it works for you. I do think, however, that you should write integration tests to empathize with your end users. Which order you write the tests in is up to you. In any case, you should be flexible in your design and refactor mercilessly.

The words of a prolific architect and inventor come to mind:

When I am working on a problem, I never think about beauty but when I have finished, if the solution is not beautiful, I know it is wrong.

—R. Buckminster Fuller

The previous quicksort example provides is an illustration of how we can improve our interface, which becomes apparent when writing tests for this library. Currently, we have a standalone quicksort() function, which accepts a slice as input. This is fine, but we can make our code more Rustaceous by creating a trait (traits are discussed in greater detail in chapter 8) and providing an implementation, as shown in the following listing.

Listing 7.4 Code for Quicksort trait

pub trait Quicksort {                                       
    fn quicksort(&mut self) {}                              
}
 
impl<T: std::cmp::PartialOrd + Clone> Quicksort for [T] {   
    fn quicksort(&mut self) {
        quicksort(self);                                    
    }
}

Here, we define our public quicksort trait.

We’ll use the quicksort() method (rather than sort()) so that we don’t clash with the existing sort() method on Vec and slices.

Here, we define a generic implementation for our trait, which will work for any slice type that implements the PartialOrd and Clone traits.

Here, we just call our quicksort implementation directly.

Now, we can update our tests, as shown in the following listing.

Listing 7.5 Code for integration test with the quicksort trait

#[test]
fn test_quicksort_trait() {
    use quicksort_trait::Quicksort;                   
 
    let mut values = vec![12, 1, 5, 0, 6, 2];
    values.quicksort();                               
    assert_eq!(values, vec![0, 1, 2, 5, 6, 12]);
 
    let mut values = vec![1, 13, 5, 10, 6, 2, 0];
    values.quicksort();                               
    assert_eq!(values, vec![0, 1, 2, 5, 6, 10, 13]);
}

All we need to import is the quicksort trait.

Instead of quicksort(&mut values), we can just write values.quicksort().

This code doesn’t look substantially different, and for the most part, we’re just using a bit of syntax sugar to clean things up. Calling arr.quicksort() instead of quicksort(&mut arr) looks nicer and requires typing four fewer characters, as we don’t need to specify the explicit mutable borrow with &mut.

7.3 Built-in integration testing vs. external integration testing

Rust’s built-in integration testing will serve most people well, but it’s not a panacea. You may benefit from external integration testing tools, from time to time. For example, testing an HTTP service in Rust could be best served with simple (and nearly ubiquitous) tools like curl (https://curl.se/) or HTTPie (https://github.com/httpie/httpie). These tools aren’t related to Rust specifically; they are generic tools that operate at the system level, rather than the language level.

A quick web search will show there are many, many existing software test tools, especially for HTTP services. Unless you’re trying to create your own testing framework, it’s almost always better to use existing tools than reinvent the wheel.

For command line applications written in Rust, writing the integration tests in Rust isn’t always the best approach. Rust is designed for safety and performance—test harnesses don’t usually need to be safe or fast, just correct. In many cases, it’ll be much easier to write integration tests as Bash, Ruby, or Python script rather than a Rust program.

While it’s great to do everything in Rust, you’ll need to weigh the value of spending the time required to build your integration tests in Rust, depending on the complexity involved. Dynamic scripting languages offer many advantages for noncritical applications, as you can usually make things happen quickly with little effort, even if you’re an expert in Rust.

However, there is one big advantage to only using Rust for integration tests: you’ll be able to run your tests on any platform supported by Rust, with no need for external tooling aside from the Rust toolchain. This can have some advantages, especially in constrained environments. Additionally, if you find Rust to be your most productive language, then there’s no reason not to use it.

7.4 Integration testing libraries and tooling

Most of the tools and libraries used for unit testing also apply for integration tests. There are, however, a few crates that can make life much easier for integration testing, which we’ll explore in this section.

7.4.1 Using assert_cmd to test CLI applications

For testing command line applications, let’s look at the assert_cmd crate (https://crates.io/crates/assert_cmd), which makes it easy to run commands and check their result. To demonstrate, we’ll create a command line interface for our quicksort implementation, which sorts integers from CLI arguments, shown in the following listing.

Listing 7.6 CLI application using quicksort

use std::env;
 
fn main() {
    use quicksort_proptest::Quicksort;
 
    let mut values: Vec<i64> = env::args()
        .skip(1)                                                         
        .map(|s| s.parse::<i64>().expect(&format!("{s}: bad input: ")))  
        .collect();                                                      
 
    values.quicksort();
 
    println!("{values:?}");
}

Reads the command line arguments, skipping the first argument, which is always the program name

Parses each value (a string) into an i64

Collects the values into a Vec

We can test this by running cargo run 5 4 3 2 1, which will print [1, 2, 3, 4, 5].

Now, let’s write some tests using assert_cmd in the following listing.

Listing 7.7 Quicksort CLI integration tests using assert_cmd

use assert_cmd::Command;
 
#[test]
fn test_no_args() -> Result<(), Box
 <dyn std::error::Error>> {                           
    let mut cmd = Command::cargo_bin("quicksort-cli")?;
    cmd.assert().success().stdout("[]\n");
 
    Ok(())                                              
}
 
#[test]
fn test_cli_well_known() -> Result<(), Box
 <dyn std::error::Error>> {                           
    let mut cmd = Command::cargo_bin("quicksort-cli")?;
    cmd.args(&["14", "52", "1", "-195", "1582"])
        .assert()
        .success()
        .stdout("[-195, 1, 14, 52, 1582]\n");
 
    Ok(())                                              
}

Our test functions return a Result, which lets us use the ? operator.

At the end of the test, we just return Ok(()). () is the special unit type, which can be used as a placeholder and has no value. It can be thought of as equivalent to a tuple with zero elements.

Our test functions return a Result, which lets us use the ? operator.

At the end of the test, we just return Ok(()). () is the special unit type, which can be used as a placeholder and has no value. It can be thought of as equivalent to a tuple with zero elements.

These tests are fine, but for testing against well-known values, we can do a little better. Rather than hardcoding into the source code, we can create some simple file-based fixtures to test against well-known values in a programmatic way.

First, we’ll create a simple directory structure on the filesystem to store our test fixtures. The structure consists of numbered folders, with a file for the arguments (args) and the expected result (expected):

$ tree tests/fixtures
tests/fixtures
├── 1
│   ├── args
│   └── expected
├── 2
│   ├── args
│   └── expected
└── 3
    ├── args
    └── expected
 
3 directories, 6 files

Next, we’ll create a test that iterates over each directory within the tree and reads the arguments and expected result; then runs our test; and, finally, checks the result, as shown in the following listing.

Listing 7.8 Quicksort CLI integration tests with file-based fixtures

#[test]
fn test_cli_fixtures() -> Result<(), Box<dyn std::error::Error>> {
    use std::fs;
    let paths = fs::read_dir("tests/fixtures")?;                 
 
    for fixture in paths {                                       
        let mut path = fixture?.path();
        path.push("args");                                       
        let args: Vec<String> = fs::read_to_string
         (&path)?                                              
            .trim()                                              
            .split(‘ ‘)                                          
            .map(str::to_owned)                                  
            .collect();                                          
        path.pop();                                              
        path.push("expected");                                   
        let expected = fs::read_to_string(&path)?;               
 
        let mut cmd = Command::cargo_bin                         
         ("quicksort-cli")?;                                   
        cmd.args(args).assert().success().stdout                 
         (expected);                                           
    }
 
    Ok(())
}

Performs a directory listing within tests/fixtures within our crate

Iterates over each listing within the directory

Pushes the args name into our path buffer

Reads the contents of the args file into a string and parses the string into a Vec—the trim() method removes the trailing newline from the args file; split(' ') will split the contents on spaces; map(str::to_owned) will convert a &str into an owned String; and, finally, collect() will collect the results into a Vec.

Pops args off of the path buffer

Pushes expected onto the path buffer

Reads the expected values from the file into a string

Runs the quicksort CLI, passes the arguments, and checks the expected results

7.4.2 Using proptest with integration tests

Next, to make our tests even more robust, we can add the proptest crate (which we discussed in the previous chapter) to our quicksort implementation within an integration test in the following listing.

Listing 7.9 Proptest-based integration test with quicksort

use proptest::prelude::*;
 
proptest! {
    #[test]
    fn test_quicksort_proptest(
        vec in prop::collection::vec(prop::num::i64::ANY, 0..1000)
    ) {                                        
        use quicksort_proptest::Quicksort;
 
        let mut vec_sorted = vec.clone();
        vec_sorted.sort();                     
 
        let mut vec_quicksorted = vec.clone();
        vec_quicksorted.quicksort();           
 
        assert_eq!(vec_quicksorted, vec_sorted);
    }
}

prop::collection::vec provides us with a Vec of random integers, with a length up to 1,000.

Here, we clone then sort (using the built-in sorting method) the random values to use as our control.

Here, we clone and sort the random values using our quicksort implementation.

It’s worth noting that testing with tools that automatically generate test data—such as proptest—can have unintended consequences, should your tests have external side effects, such as making network requests or writing to an external database. You should try to design your tests to account for this, either by setting up and tearing down the whole environment before and after each test or providing some other way to return to a known good state before and after the tests run. You may discover some surprising edge cases when using random data.

Note The proptest crate prints the following warning when running as an integration test: proptest: FileFailurePersistence::SourceParallel set, but failed to find lib.rs or main.rs. This warning can be ignored; refer to the GitHub issue at https://github.com/AltSysrq/proptest/issues/233 for more information.

7.4.3 Other integration testing tools

The following are some more crates worth mentioning to turbocharge your integration tests:

7.5 Fuzz testing

Fuzz testing is similar to property testing, which we’ve already discussed in this chapter. The difference between the two, however, is that with fuzz testing, you test your code with randomly generated data that isn’t necessarily valid. When we do property testing, we generally restrict the set of inputs to values we consider valid. With property testing, we do this because it often doesn’t makes sense to test all possible inputs, and we also don’t have infinite time to test all possible input combinations.

Fuzz testing, on the other hand, does away with the notion of valid and invalid and simply feeds random bytes into your code, so you can see what happens. Fuzz testing is especially popular in security-sensitive contexts, where you want to understand what happens when code is misused.

A common example of this is public-facing data sources, such as web forms. Web forms can be filled with data from any source that needs to be parsed, validated, and processed. Since these forms are out in the wild, there’s nothing stopping someone from filling web forms with random data. For example, imagine a login form with a username and password, where someone (or something) could try every combination of username and password, or a list of the most common combinations, to gain access to the system, either by guessing the correct combination or injecting some “magic” set of bytes that causes an internal code failure and bypasses the authentication system. These types of vulnerabilities are surprisingly common, and fuzz testing is one strategy to mitigate them.

The main problem with fuzz testing is that it can take an unfeasible amount of time to test every set of possible inputs, but in practice, you don’t necessarily need to test every combination of input bits to find bugs. You may be quite surprised how quickly a fuzz test can find bugs in code you may have thought was bulletproof.

To fuzz test, we’re going to use a library called libFuzzer (https://llvm.org/docs/LibFuzzer.html), which is part of the LLVM project. You could use libFuzzer directly with FFI (we explored FFI in chapter 4), but instead, we’ll use a crate called cargo-fuzz, which takes care of providing a Rust API for libFuzzer and generates boilerplate for us.

Before we dive into a code sample, let’s talk about how libFuzzer works at a high level: the library will populate a structure (which you provide) with random data that contains function arguments, and it calls your code’s function repeatedly. If the data triggers an error, this is detected by the library, and a test case is constructed to trigger the bug.

Once we’ve installed the cargo-fuzz crate with cargo install cargo-fuzz, we can write a test. In the following listing, I’ve constructed a relatively simple function that looks like it works, but in fact, it contains a subtle bug that will trigger under specific conditions.

Listing 7.10 String-parsing function with a bug

pub fn parse_integer(s: &str) -> Option<i32> {                            
    use regex::Regex;
    let re = Regex::new(r"^-?\d{1,10}$").expect("Parsing regex failed");  
    if re.is_match(s) {
        Some(s.parse().expect("Parsing failed"))
    } else {
        None
    }
}

Checks if string contains _only_ digits using a regular expression, including negative numbers

Will match a string with 1-10 digits, prefixed by an option "-"

This function will accept a string as input and parse the string into an i32 integer, provided it’s between 1 and 10 digits in length and, optionally, prefixed by a - (minus) symbol. If the input doesn’t match the pattern, None is returned. The function shouldn’t cause our program to crash on invalid input. This seems innocuous enough, but there’s a major bug.

These kinds of bugs are surprisingly common and something we have all probably written at some point. Edge case bugs like this can lead to undefined behavior, which—in a security context—can lead to bad things happening.

Next, let’s create a small fuzz test using cargo-fuzz. First, we need to initialize the boilerplate code by running cargo fuzz init. This will create the following structure within our project:

$ tree .
.
├── Cargo.lock
├── Cargo.toml
├── fuzz
│   ├── Cargo.lock
│   ├── Cargo.toml
│   └── fuzz_targets
│       └── fuzz_target_1.rs
└── src
    └── lib.rs
 
3 directories, 6 files

Here, we can see that cargo-fuzz created a new project in the fuzz subdirectory, and there’s a test in fuzz_target_1. We can list the fuzz targets (or tests) with cargo fuzz list, which will print fuzz_target_1.

Next, we need to write the fuzz test. To test our function, we’re just going to call it with a random string, which is supplied by the fuzzing library. We’ll use the Arbitrary (https://crates.io/crates/arbitrary) crate to derive data in the form we need. The fuzz test is shown in the following listing.

Listing 7.11 Fuzz test

#![no_main]
use arbitrary::Arbitrary;
use libfuzzer_sys::fuzz_target;
 
#[derive(Arbitrary, Debug)]         
struct Input {
    s: String,                      
}
 
fuzz_target!(|input: Input| {       
    use fuzzme::parse_integer;
 
    parse_integer(&input.s);        
});

We use derive to automatically generate the Arbitrary and Debug traits.

Our Input struct only contains one string, nothing else. The data in this struct will be populated arbitrarily by the fuzzer.

Here, we use the fuzz_target! macro provided by cargo-fuzz, which defines the entrypoint for our fuzz test.

Here, we call our function with our random string data, which is provided by the fuzzer.

Now, we’re ready to run the fuzz test and see what happens. You probably already know at this point that there’s a bug, so we expect it to crash. When we run the fuzzer with cargo fuzz run fuzz_target_1, we’ll see output that looks similar to the following listing (which has been shortened because the fuzzer generates a lot of logging output).

Listing 7.12 Output of cargo fuzz run fuzz_target_1

cargo fuzz run fuzz_target_1
   Compiling fuzzme-fuzz v0.0.0
   (/Users/brenden/dev/code-like-a-pro-in-rust/code/c7/7.5/fuzzme/fuzz)
    Finished release [optimized] target(s) in 1.07s
    Finished release [optimized] target(s) in 0.01s
     Running `fuzz/target/x86_64-apple-darwin/release/fuzz_target_1
     -artifact_prefix=/Users/brenden/dev/code-like-a-pro-in-rust/
      code/c7/7.5/fuzzme/fuzz/artifacts/fuzz_target_1/
/Users/brenden/dev/code-like-a-pro-in-rust/code/c7/7.5/fuzzme/
 fuzz/corpus/fuzz_target_1`
fuzz_target_1(14537,0x10d0a6600) malloc: nano zone abandoned due to
inability to preallocate reserved vm space.
INFO: Running with entropic power schedule (0xFF, 100).
... snip ...
 
Failing input:
 
    fuzz/artifacts/fuzz_target_1/
    crash-105eb7135ad863be4e095db6ffe64dc1b9a1a466
 
Output of `std::fmt::Debug`:
 
    Input {
        s: "8884844484",
    }
 
Reproduce with:
 
    cargo fuzz run fuzz_target_1 fuzz/artifacts/fuzz_target_1/
    crash-105eb7135ad863be4e095db6ffe64dc1b9a1a466
 
Minimize test case with:
 
    cargo fuzz tmin fuzz_target_1 fuzz/artifacts/fuzz_target_1/
    crash-105eb7135ad863be4e095db6ffe64dc1b9a1a466

Note Running the fuzzer can take quite some time, even on fast machines. While this example should trigger fairly quickly (within 60 seconds, in most cases), more complex tests may take much longer. For unbounded data (e.g., a string with no limit in length), the fuzzer can take an infinite amount of time.

Near the bottom of the output, cargo-fuzz prints information about the input that caused the crash. Additionally, it creates a test case for us, which we can use to make sure this bug isn’t triggered again in the future. For the preceding example, we can simply run cargo fuzz run fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-105eb7135ad863be4e095db6ffe64dc1b9a1a466 to test our code again with the same input, which will make it easy to test a fix for this bug without having to rerun the fuzzer from scratch. It can take a long time to find test cases that trigger a crash, so this is helpful in limiting the amount of time we need to spend running the fuzzer.

As an exercise, try modifying the function so that it no longer crashes. There are a few different ways to solve this problem, and I’ll provide a hint: the parse() method already returns a Result for us. For more details on using cargo-fuzz, consult the documentation at https://rust-fuzz.github.io/book/.

Summary