Have you ever written a unit test with many asserts in it?
I know I have.
I’ve done it because I wanted to make sure that the code I was testing was doing the right thing. If you’ve done this too, you know that it often makes it difficult to see the errors.
What are multiple asserts, and is it a bad thing to use them in unit testing? Multiple asserts occur when you use more than one assert statement in a single unit test. This is bad when you use multiple assert statements to check various scenarios and make the test function more complicated to read and maintain. Multiple asserts are good if you use them to check the multiple properties of a single object.
What are the pros and cons of having multiple asserts in a single unit test? This article will cover the topic in-depth and will answer this question.
Multiple asserts per test – explanation
Unit testing is a great way to verify that your code is working as expected. By writing unit tests, you can check that the production code works as expected. And usually, one test method only has one assert at the end of it.
But what happens if you have multiple asserts in a unit test? If you have more than one assert in a unit test, the unit test will be completed successfully if all assertions are met.
When writing a good unit test, one of the golden rules is to include as few asserts as possible, to make the test more readable, and therefore more maintainable.
When it comes to multiple asserts in a unit test, there is a wrong and correct way to use them.
Using multiple asserts – the wrong way
So if you’re writing a test case for a method and want to check multiple scenarios, the best way is to write multiple asserts within the test, right? Wrong.
When it comes to testing, the right way often isn’t the easy way. Using multiple assertions in a single test means that your test is more complicated and will be harder to get right.
The wrong way of using the multiple asserts in your test function:
- arrange the system under test
- act, call the method you want to check
- assert the output
- for the same system under test, call another method
- assert the output
- call the same method, using a different scenario and different input parameters
- assert the result
The equivalent is this unit test case example:
[Fact]
public void test_AccountService_works_correctly()
{
//1
var bookServiceStub = new Mock<IBookService>();
var emailSenderMock = new Mock<IEmailSender>();
bookServiceStub
.Setup(x => x.GetBooksForCategory("UnitTesting"))
.Returns(new List<string>
{
"The Art of Unit Testing",
"Test-Driven Development",
"Working Effectively with Legacy Code"
});
bookServiceStub
.SetupSequence(x => x.GetISBNFor(It.IsAny<string>()))
.Returns("0-9020-7656-6")
.Returns("0-9180-6396-5")
.Returns("0-3860-1173-7")
.Returns("0-5570-1450-6");
//2
var accountService = new AccountService(bookServiceStub.Object, emailSenderMock.Object);
//3
IEnumerable<string> result = accountService.GetAllBooksForCategory("UnitTesting");
//4
Assert.Equal(3, result.Count());
//5
string result2 = accountService.GetBookISBN("UnitTesting", "art");
//6
Assert.Equal("0-9020-7656-6", result2);
//7
accountService.SendEmail("test@gmail.com", "Test - Driven Development");
//8
emailSenderMock.Verify(x => x.SendEmail(
"test@gmail.com",
"Awesome Book",
$"Hi,\n\nThis book is awesome: Test - Driven Development.\nCheck it out."), Times.Once);
}
The test does the following:
- It sets up dependencies using Moq
- It creates the system under test,
AccountService - The first act, calls the
GetAllBooksForCategorymethod - The first assert – check the output of the
GetAllBooksForCategorymethod - The second act, calls the
GetBookISBNmethod - The second assert – check the output of the
GetBookISBNmethod - The third act – call the
SendEmailmethod - Use mock to verify that the email was sent
As you might expect, that unit test takes too long to write. It is hard to understand what is going on in the test and even harder to know at first what’s the problem if a test fails.
The right approach is to split that test into several tests.
How to use multiple asserts correctly
Multiple asserts are good if you are testing more than one property of an object simultaneously. This could happen because you have two or more properties on an object that are related. You should use multiple asserts in this case so that all the tests on that object fail if any one of them fails.
BookDocument bookDocument = accountService.GetBookDocumentForISBN("9780321146533");
Assert.Equal("9780321146533", bookDocument.ISBN);
Assert.Equal('3', bookDocument.ISBNCheckDigit);
Assert.Equal("Kent Beck", bookDocument.Author);
Assert.Equal("Test Driven Development: By Example", bookDocument.Name);
In this example, the test is checking the BookDocument result of the Test Driven Development book. It checks that all properties of this book are correctly populated.
When to use multiple asserts?
1. If you are testing more than one property of an object at the same time. – You might want to assert against more than one property of a class at the same time when the properties are related, as in the test above. Those properties represent a part of a single object, and if one is not working properly, it will be reflected in another.
2. If you first want to check that the property is not null and after that the property value.
The advantages of multiple asserts per test
The biggest advantage of having multiple asserts per test is that you can reuse the same code of the arrange part of the test. Let’s say you have multiple test cases that are checking the same class, but the class is complicated to set up. In that case, you can decrease the number of lines in your test file by having multiple assertions.
If that’s the case with your code, you need to ask yourself: Am I covering the symptoms of a disease with multiple asserts? The disease, in this case, is a complicated class that has too many responsibilities and too many dependencies. That’s why the arrange part of the test has many parts.
The other way to tackle complicated setup and avoid duplication in the test method is to use the Extract Method refactoring. If you have a test that uses multiple asserts, extract your common setup into a separate method. Then split the existing test into multiple test methods.
How to have meaningful multiple asserts in a test?
When developers use assertions, they usually focus on the positive cases, where the assertion succeeds. But in reality, it is much more common that an assertion will fail. In xUnit, the exception is thrown on a first failed assertion. This means all other asserts won’t be checked after that. Therefore, you will only get the first assertion error in the test runner output window.
So, how do you handle multiple assertions when they don’t work out the way you expected? Let see what the alternatives are in this case.
AssertAll custom class – for xUnit and MSTest
You can create a custom AssertAll class with the Check static method:
public class AssertAll
{
public static void Check(params Action[] assertions)
{
var errorMessages = new List<string>();
foreach (var action in assertions)
{
try
{
action.Invoke();
}
catch (Exception ex)
{
errorMessages.Add(ex.Message);
}
}
if (errorMessages.Count == 0)
{
return;
}
string errorMessageString = string.Join(Environment.NewLine, errorMessages);
throw new XunitException($"The following conditions failed: {Environment.NewLine}{errorMessageString}");
}
}
The method Check takes variable number of actions. It then executes every action. In case of any exception, the exception is stored into a list and will form a part of the final thrown exception.
The example of usage:
BookDocument bookDocument = accountService.GetBookDocumentForISBN("9780321146533");
AssertAll.Check
(
() => Assert.Equal("9780321146533", bookDocument.ISBN),
() => Assert.Equal('3', bookDocument.ISBNCheckDigit)
);
With this class, you can get the output of multiple assertions in the test result window.
This approach works for xUnit and MSTest frameworks, while the NUnit testing framework already has this built-in.
Multiple asserts – built-in in NUnit
Like I mentioned above, the NUnit test framework has a way to evaluate multiple assertions at once.
Example:
[Test]
public void MultipleAssertTest()
{
BookDocument bookDocument = accountService.GetBookDocumentForISBN("9780321146533");
Assert.Multiple(() =>
{
Assert.AreEqual("9780321146533", bookDocument.ISBN);
Assert.AreEqual('3', bookDocument.ISBNCheckDigit);
});
}
The Assert.Multiple can also contain any other code, not just assert statements.
Should you use multiple or single assert in integration testing?
Integration testing is an essential type of software testing. These automated tests check if the various parts of an application work together and interact as expected. You usually write your integration test suite at the end of the development cycle after each module of the application is developed separately.
Of the many challenges facing a developer, integration testing is one of the hardest.
It can be challenging to test a large system in a controlled manner. This is partly because integration tests require a lot of setup:
- You need to set up the clean test database pre-populated with initial data
- You need to set up the proper network endpoint that you will call in the test code
- You need to prepare all necessary files and folders
- You need to make a proper clean up at the end of the test so that the testing environment stays clean for other tests
Developers must write code covering multiple components in a large system and then run it through multiple tests to ensure that everything integrates properly. The amount of work required to get a comprehensive integration test suite for a large system is a lot, and it is something that will often take multiple developers and multiple months. One way to ease this process is to use multiple assertions in your integration tests.
For example:
- At the beginning of your test, you can assert that the database contains the correct initial data
- You can assert that the API endpoint is reachable
- You can assert that the necessary files exist
By doing so, you will get the proper test result if any of the preconditions are not satisfied. This will make your integration test more stable.
When it comes to unit testing, the preference is to use a single assert method. But when it comes to integration testing, using multiple asserts can help you better understand what is going on in the testing environment.
Should you use multiple or single asserts in UI (end-to-end) testing?
In UI testing, assertions are statements that validate the behavior of the UI under test.
An assertion can be positive or negative. A positive assertion checks that the expected result occurs; a negative assertion checks that an unexpected result doesn’t occur.
Multiple assertions are typically used in UI testing to validate the UI under the test at different levels of detail. For example, an assertion can check that the user can click on the “Sign in” button and lead to the “Sign in” page.
The other reason to use multiple asserts is speed. As you know, the UI test is the slowest possible test type. To increase the whole UI test project execution speed, you can do multiple asserts for a page once you reach it within an application.
Conclusion
When I first learned what assertions were, I was told that they should only be used once per test.
I learned that this was because assertions are a way to verify that your code is working, and the more assertions you have, the less understandable your test is. However, multiple asserts shouldn’t be avoided all the time.
Use them while testing several properties of a single class.
Use them while writing an integration test or UI test. They can improve speed and stability.
But don’t use them in unit tests to check multiple scenarios in one test method.