Writing tests is a lot like writing code – you must follow a specific process to ensure your code is working properly. Sometimes, you need to run them in a specific order because of circumstances. That way, you can ensure that they pass in the desired order.
This article will discuss three different ways to run tests in a specific order in Visual Studio.
Which approach you will take depends on your unit testing framework (MSTest, NUnit, xUnit). The last approach is unit testing framework agnostic, meaning it can work with any framework.
How to control unit test order execution in Visual Studio 2022?
If you’re a software developer, you know that writing and running tests is an important part of the development process.
But did you know that you can also control the order in which your tests run?
The way to control test order execution depends on your unit testing framework. It can be:
However, before we start, it’s important to have something in mind. It is not one of the unit testing best practices to order unit tests. Ideally, unit test methods should not be interdependent.
The term “test dependency” is often used to describe how one unit test is dependent on the outcome of another. But, this is problematic because the whole suite of tests can fail as soon as one test fails. To avoid this problem, you should avoid test interdependence in your unit tests.
Ideally, you would want to allow unit test cases to run in any order. If you run them in parallel, the order becomes less important, and you get a shorter test suite execution time.
That being said, if you have a scenario where you need to run tests in a specific order, here are the options.
How to run tests in a specific order with MSTest?
MSTest is a unit testing framework that comes bundled with Visual Studio, and, for the most part, it can do a decent job for unit testing.
MSTest automatically orders tests by their test name, making it easy to find and run the ones you need. So, to control test order execution with MSTest, the first thing you can do is to prefix your unit test methods and control the order that way.
For example, prefix your tests with TXXXXX, where is the test number:
- T00001_test_something
- T00002_test_something_else
- T00003_third_test_to_be_executed
However, with this approach, you must pay attention to how you name tests and ensure you or another developer doesn’t duplicate the test name prefix. So, this is not an ideal option.
Next, you can use a custom helper class to help you control the execution of tests.
public static class TestExecution
{
public static void RunTestsInOrder(params Action[] testMethods)
{
List<Exception> exceptions = new List<Exception>();
foreach (var test in testMethods)
{
try
{
test();
}
catch (Exception ex)
{
exceptions.Add(ex);
}
}
if (exceptions.Count > 0)
{
throw new Exception(
string.Join("; ", exceptions.Select(e => e.Message),
new AggregateException(exceptions))
);
}
}
}And the usage:
[TestMethod]
public void RunBatchOfTests()
{
TestExecution.RunTestsInOrder(
First_test,
Second_test,
Third_test
);
}
//Important: omit the TestMethod attribute
public void First_test()
{
Assert.Fail();
}
//Important: omit the TestMethod attribute
public void Second_test()
{
}
//Important: omit the TestMethod attribute
public void Third_test()
{
Assert.Fail();
}This way, all tests execute in the correct order. You get an error indicating which tests failed if one or more tests fail.
The important thing here is to omit the [TestMethod] attribute from the First_test(), Second_test() and the Third_test(). That way, they will only execute within the RunBatchOfTests test method and in the order you specify within the test case.
The above is the suggested way to group test methods within one single test method. If you specify all methods like this:
[TestMethod]
public void WrongWayToGroupTests()
{
First_test();
Second_test();
Third_test();
}then the whole test execution will stop after the first test fails.
The third option is to use the TestProperty method attribute:
[TestClass]
public class UnitTest2
{
public static bool Test1Called;
public static bool Test2Called;
public static bool Test3Called;
[TestMethod]
[TestProperty("ExecutionOrder", "1")]
public void First_test()
{
Test1Called = true;
Assert.IsFalse(Test2Called);
Assert.IsFalse(Test3Called);
}
[TestMethod]
[TestProperty("ExecutionOrder", "3")]
public void Second_test()
{
Test2Called = true;
Assert.IsTrue(Test1Called);
Assert.IsTrue(Test3Called);
}
[TestMethod]
[TestProperty("ExecutionOrder", "2")]
public void Third_test()
{
Test3Called = true;
Assert.IsTrue(Test1Called);
Assert.IsFalse(Test2Called);
}
}and then use the “Group by” button in the Visual Studio Test Explorer window and sort by Trait.
How to run tests in a specific order with xUnit?
xUnit is another unit testing framework that can run tests in different ways.
To control test order execution with xUnit, you have two options:
1. Implement the ITestCaseOrderer and provide implementation to control the test order. The following implementation will order unit tests by their name.
internal class AlphabeticalTestOrderer : ITestCaseOrderer
{
/// <summary>
/// Orders test cases for execution.
/// </summary>
/// <param name="testCases">The test cases to be ordered.</param>
/// <returns>The test cases in the order to be run.</returns>
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases)
where TTestCase : ITestCase
{
return testCases.OrderBy(testCase => testCase.TestMethod.Method.Name);
}
}Then, use the attribute to decorate the class.
[TestCaseOrderer(
"RunTestsInOrder.XUnit.AlphabeticalTestOrderer",
"RunTestsInOrder.XUnit")]
public class UnitTest1
{
[Fact]
public void First_test()
{
}
[Fact]
public void Second_test()
{
}
[Fact]
public void Third_test()
{
}
}2. Specify a custom attribute to control the execution order (implementation from Microsoft Learn document).
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestPriorityAttribute : Attribute
{
public int Priority { get; private set; }
public TestPriorityAttribute(int priority) => Priority = priority;
}Next, implement a new test orderer that will take into account the above attribute:
public class PriorityOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(
IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
{
string assemblyName = typeof(TestPriorityAttribute).AssemblyQualifiedName!;
var sortedMethods = new SortedDictionary<int, List<TTestCase>>();
foreach (TTestCase testCase in testCases)
{
int priority = testCase.TestMethod.Method
.GetCustomAttributes(assemblyName)
.FirstOrDefault()
?.GetNamedArgument<int>(nameof(TestPriorityAttribute.Priority)) ?? 0;
GetOrCreate(sortedMethods, priority).Add(testCase);
}
foreach (TTestCase testCase in
sortedMethods.Keys.SelectMany(
priority => sortedMethods[priority].OrderBy(
testCase => testCase.TestMethod.Method.Name)))
{
yield return testCase;
}
}
private static TValue GetOrCreate<TKey, TValue>(
IDictionary<TKey, TValue> dictionary, TKey key)
where TKey : struct
where TValue : new() =>
dictionary.TryGetValue(key, out TValue? result)
? result
: (dictionary[key] = new TValue());
}If you run the following tests:
[TestCaseOrderer(
"RunTestsInOrder.XUnit.PriorityOrderer",
"RunTestsInOrder.XUnit")]
public class UnitTest2
{
[Fact, TestPriority(3)]
public void First_test()
{
Debug.WriteLine("First_test");
}
[Fact, TestPriority(2)]
public void Second_test()
{
Debug.WriteLine("Second_test");
}
[Fact, TestPriority(1)]
public void Third_test()
{
Debug.WriteLine("Third_test");
}
}in the Debug mode:
you will see the following messages in the Output window inside Visual Studio 2022:
How to run tests in a specific order with NUnit?
NUnit is another popular unit testing framework that supports test cases with dependencies on other tests.
NUnit provides the easiest way to control test order execution. It has the OrderAttribute that you can put on a test method.
public class Tests
{
[Test, Order(3)]
public void First_test()
{
TestContext.Out.WriteLine("First_test");
}
[Test, Order(2)]
public void Second_test()
{
TestContext.Out.WriteLine("Second_test");
}
[Test, Order(1)]
public void Third_test()
{
TestContext.Out.WriteLine("Third_test");
}
}The important note here is the test runner will execute test methods with the OrderAttribute before the test method without this attribute.
The above tests use the TextContext class to write to the Output window. In case you want to know more about writing to the console and the Output window in Visual Studio in unit tests, check out this article.
Conclusion
Depending on your unit testing framework, there are several ways to run tests in a specific order. For example, with MSTest, you can control test execution order by prefixing your test methods with certain letters or numbers. Alternatively, you can use a custom helper class to help specify the test order in your test case.
For xUnit, you can use ITestCaseOrderer implementation. And for NUnit, you can use the OrderAttribute to control the test execution order.
Ultimately, which approach you use will depend on your specific testing requirements.
So if you need to run tests in a particular order for your project, explore the options available and choose the one that works best for you.