Skip to content
Tutorials Center
My learning

Topics

Lesson 154 of 182 ยท Testing and Quality

xUnit, NUnit and MSTest

Compare the three .NET test frameworks - attributes, assertions, theories and data-driven tests, fixtures and lifecycle - with read-only real code and runnable stand-ins you can experiment with.

In the previous lesson you built a test runner of about forty lines. Real projects do not do that. They use one of three mature frameworks: xUnit, NUnit or MSTest. All three find test methods by reflection, run them in isolation, report failures and plug into dotnet test, Visual Studio, Rider, VS Code and every build server. The concepts are almost identical; only the attribute names and a few defaults differ.

There is one practical limit in this tutorial. The framework packages are NuGet packages, and our online editor cannot download packages or run dotnet test. So each idea appears twice:

  1. as read-only code you would put in a real test project, labelled with what you would see on your machine, and
  2. as a runnable stand-in: a miniature framework with the same attribute names, so you can run, change and break the examples right here.

The runnable framework in this lesson is a teaching model. It copies the shape of xUnit (attributes, constructor-per-test, fixtures) but none of its code or its full feature set. Use the real packages in real projects.

Setting up a test project

A test project is an ordinary class library with a test framework and a runner adapter referenced, plus a reference to the project under test. You create it with the dotnet CLI (shown read-only because our sandbox has no CLI):

Terminal
# On your machine, not in our editor
dotnet new sln -n Shop
dotnet new classlib -o src/Shop.Core
dotnet new xunit   -o tests/Shop.Core.Tests
dotnet sln add src/Shop.Core tests/Shop.Core.Tests
dotnet add tests/Shop.Core.Tests reference src/Shop.Core
dotnet test

The templates are xunit, nunit and mstest. The generated .csproj looks like this (package versions change often, so check NuGet for current ones):

XML
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="..." />
    <PackageReference Include="xunit" Version="..." />
    <PackageReference Include="xunit.runner.visualstudio" Version="..." />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\..\src\Shop.Core\Shop.Core.csproj" />
  </ItemGroup>
</Project>

Microsoft.NET.Test.Sdk is the glue that makes dotnet test work, and the runner adapter (xunit.runner.visualstudio, NUnit3TestAdapter, MSTest.TestAdapter) lets tools discover tests.

The same test in three frameworks

Let us test a small class: a PasswordRules helper that says whether a password is acceptable. First xUnit, the most popular choice for new .NET code (and the framework Microsoft itself uses for the .NET runtime repositories):

C#
// Read-only: needs the xunit NuGet package
using Xunit;

public class PasswordRulesTests
{
    [Fact]
    public void Accepts_a_long_password_with_a_digit()
    {
        bool ok = PasswordRules.IsAcceptable("correct-horse-7");

        Assert.True(ok);
    }

    [Theory]
    [InlineData("short1")]
    [InlineData("nodigitsatall")]
    [InlineData("")]
    public void Rejects_weak_passwords(string password)
    {
        Assert.False(PasswordRules.IsAcceptable(password));
    }
}

The same tests in NUnit:

C#
// Read-only: needs the NUnit and NUnit3TestAdapter NuGet packages
using NUnit.Framework;

[TestFixture]
public class PasswordRulesTests
{
    [Test]
    public void Accepts_a_long_password_with_a_digit()
    {
        bool ok = PasswordRules.IsAcceptable("correct-horse-7");

        Assert.That(ok, Is.True);
    }

    [TestCase("short1")]
    [TestCase("nodigitsatall")]
    [TestCase("")]
    public void Rejects_weak_passwords(string password)
    {
        Assert.That(PasswordRules.IsAcceptable(password), Is.False);
    }
}

And in MSTest:

C#
// Read-only: needs the MSTest NuGet packages
using Microsoft.VisualStudio.TestTools.UnitTesting;

[TestClass]
public class PasswordRulesTests
{
    [TestMethod]
    public void Accepts_a_long_password_with_a_digit()
    {
        bool ok = PasswordRules.IsAcceptable("correct-horse-7");

        Assert.IsTrue(ok);
    }

    [TestMethod]
    [DataRow("short1")]
    [DataRow("nodigitsatall")]
    [DataRow("")]
    public void Rejects_weak_passwords(string password)
    {
        Assert.IsFalse(PasswordRules.IsAcceptable(password));
    }
}

Running dotnet test prints a summary similar to this (the exact wording depends on the version):

TEXT
Passed!  - Failed: 0, Passed: 4, Skipped: 0, Total: 4, Duration: 21 ms

Four tests ran: one [Fact] and three data rows. Each row of a data-driven test counts as its own test, with its own pass or fail result.

Vocabulary side by side

Idea xUnit NUnit MSTest
Mark a test class none needed [TestFixture] (optional) [TestClass]
Mark a test method [Fact] [Test] [TestMethod]
Data-driven test [Theory] + [InlineData] [TestCase] [DataRow]
Run before each test constructor [SetUp] [TestInitialize]
Run after each test IDisposable.Dispose [TearDown] [TestCleanup]
Once per class IClassFixture<T> [OneTimeSetUp] [ClassInitialize]
Skip a test [Fact(Skip = "reason")] [Ignore("reason")] [Ignore]
Expect an exception Assert.Throws<T>(...) Assert.Throws<T>(...) Assert.ThrowsExactly<T>(...)

Pick whichever your team already uses. For a new project, xUnit is a common default, NUnit has a very rich assertion and parameterization model, and MSTest has the best out-of-the-box integration with Visual Studio and has improved a lot in recent versions. Whatever you choose, learn one well; switching later is mostly a search-and-replace.

The assertion library is separate from the runner. Many teams add an assertion package such as FluentAssertions or Shouldly to any of the three frameworks for readable failures. Check the license terms before adding any of them, because they have changed over time.

Assertions

An assertion compares an actual value with an expectation and throws a special exception when they differ. The framework catches the exception and marks the test failed. The most common ones:

  • Equality: Assert.Equal(expected, actual). Note the order: expected first. Failure messages say "Expected: X, Actual: Y", so swapping them gives misleading reports.
  • Booleans and nulls: Assert.True, Assert.False, Assert.Null, Assert.NotNull.
  • Collections: Assert.Contains, Assert.Empty, Assert.Single, Assert.Equal on two sequences (element by element).
  • Exceptions: Assert.Throws<ArgumentException>(() => ...) passes only if exactly that exception type (not a derived one) is thrown, and returns the exception so you can check its message or ParamName.
  • Floating point: Assert.Equal(0.3, a + b, precision: 10), because 0.1 + 0.2 is not exactly 0.3 (see floating-point numbers).

Let us build the pieces of a mini framework so you can see how assertions and discovery fit together. This is the core: three attributes, an assertion class and a runner that understands [Fact], [Theory] and [InlineData]:

C# A tiny xUnit-style framework

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The structure matches the real thing. A [Theory] is one method with many rows of data; the runner calls it once per [InlineData] attribute and reports each call separately. The last row is a deliberate mistake: that password is acceptable, so Assert.False fails and the report shows Expected False, Actual True for just that row while the others pass.

Data-driven tests

Theories remove copy-paste. Instead of five near-identical tests, you write the logic once and list the cases. Three sources of data exist in every framework:

  • Inline values ([InlineData], [TestCase], [DataRow]): the simplest option, limited to values that can appear in an attribute (numbers, strings, null, types, enums).
  • A member that returns rows ([MemberData] in xUnit, [TestCaseSource] in NUnit, [DynamicData] in MSTest): a static method or property returning a collection of arrays, so you can pass any object, such as DateTime, records or a list.
  • A custom data attribute you write yourself, for example one that reads cases from a JSON file.

Read-only xUnit, showing the member-based form:

C#
// Read-only: needs the xunit NuGet package
public class ShippingTests
{
    public static IEnumerable<object[]> Cases =>
    [
        [new Order(Weight: 0.5, Country: "DE"), 4.90m],
        [new Order(Weight: 2.0, Country: "DE"), 7.90m],
        [new Order(Weight: 2.0, Country: "US"), 19.90m],
    ];

    [Theory]
    [MemberData(nameof(Cases))]
    public void Calculates_the_price(Order order, decimal expected)
    {
        Assert.Equal(expected, Shipping.Price(order));
    }
}

Here is the same idea running in our mini framework. To keep it short, the "member data" is a static property the runner is told to read by name:

C# Data-driven tests with objects as rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Each row is reported by its own name, so when one fails you know which input caused it. This is why good frameworks print the arguments in the test name.

Lifecycle, setup and fixtures

Three questions decide how tests share state:

  1. How often is the test class created? In xUnit, a new instance for every test method, so constructor code is "setup before each test" and IDisposable.Dispose is "cleanup after each test". NUnit and MSTest use [SetUp]/[TestInitialize] methods, and by default NUnit shares one instance per fixture (which is why fields must be reset in setup), whereas xUnit's one-instance-per-test makes shared state bugs rare.
  2. What if setup is expensive? Starting a database or web server per test would take forever. Use a fixture: an object created once and shared by all tests in a class (IClassFixture<T> in xUnit) or across classes (ICollectionFixture<T>, collections).
  3. Do tests run in parallel? xUnit runs test classes in parallel by default, but the tests inside one class sequentially. NUnit and MSTest are sequential by default and can be told to run in parallel. Anything shared and mutable needs locking or isolation.

A realistic xUnit fixture, read-only:

C#
// Read-only: needs the xunit NuGet package
public class DatabaseFixture : IDisposable
{
    public Database Db { get; } = Database.CreateInMemory(); // once per test class

    public void Dispose() => Db.Dispose();
}

public class CustomerTests : IClassFixture<DatabaseFixture>, IDisposable
{
    private readonly DatabaseFixture _fixture;

    public CustomerTests(DatabaseFixture fixture)   // runs before EACH test
    {
        _fixture = fixture;
        _fixture.Db.BeginTransaction();
    }

    [Fact]
    public void Can_add_a_customer()
    {
        _fixture.Db.Insert(new Customer("Ana"));

        Assert.Equal(1, _fixture.Db.Count<Customer>());
    }

    public void Dispose() => _fixture.Db.Rollback();    // runs after EACH test
}

The expensive database is created once. Each test starts a transaction and rolls it back, so tests stay isolated without paying the setup cost. To see the order of events, here is a runnable model of "fixture once, instance per test":

C# Fixture once, instance per test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Read the output: the fixture's setup appears once at the top, the constructor and Dispose frame each test, the instance counter is always 1, and the shared counter grows to 2. That asymmetry is the main thing to remember. Put cheap state in fields (fresh each test), put expensive read-only resources in fixtures, and be very careful about mutating fixture state, because it leaks between tests.

Async tests and exceptions

Test methods can be async Task (never async void: the runner cannot wait for it, so failures vanish). Exception assertions have async twins such as Assert.ThrowsAsync<T>. Our runner can model this too by awaiting the returned task:

C# Awaiting async tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The third test fails because it awaited a call that throws and never expected it, so the exception escapes. The real Assert.ThrowsAsync<ArgumentOutOfRangeException>(() => api.GetNameAsync(-5)) wraps the try/catch dance for you. See async and await for why tests must return Task.

Choosing and using a framework well

  • Name tests for behavior, not for methods: Rejects_passwords_without_digits beats TestIsAcceptable2.
  • One reason to fail per test. A theory row is fine; ten unrelated asserts in one test hide which behavior broke.
  • Keep tests independent. If test B needs test A to run first, merge them or extract shared setup.
  • Do not catch exceptions in tests to "make them pass"; assert them with Throws.
  • Run tests in CI on every push with dotnet test. A red build should be treated as a stop sign.
  • Use [Fact(Skip = "reason")] sparingly, always with a reason and a ticket, otherwise the skipped list only grows.
  • Integration tests for ASP.NET Core can use WebApplicationFactory<TProgram> (from the Microsoft.AspNetCore.Mvc.Testing package) to start the app in-process; see minimal APIs for the app you would test.

Never use async void for a test method and never block on tasks with .Result or .Wait() inside tests. The first hides failures, the second can deadlock in some frameworks and always hides the original exception behind an AggregateException.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In xUnit, what is the equivalent of NUnit's [SetUp] method?

    xUnit creates a new instance of the test class for every test, so the constructor is the per-test setup and IDisposable.Dispose is the per-test teardown.
  2. 2. What does a [Theory] with three [InlineData] attributes produce?

    Each row is discovered and reported as its own test case, so a failing row identifies the exact input that broke.
  3. 3. Why should an asynchronous test method return Task instead of void?

    async void gives the caller nothing to await. The test would appear to finish before it did its work, and exceptions are thrown on a different context.
  4. 4. When is a fixture (such as IClassFixture<T>) the right tool?

    Fixtures amortize costly setup such as a database or server. Because the object is shared, tests must not leave changes in it that affect other tests.
  5. 5. Which call correctly states an expected value in xUnit, so failure messages read the right way round?

    The expected value comes first, then the actual value. Swapping them still passes or fails correctly but makes the failure report misleading.

Summary

  • xUnit, NUnit and MSTest do the same job with different names: [Fact]/[Test]/[TestMethod], [Theory]/[TestCase]/[DataRow].
  • Test projects reference the framework, Microsoft.NET.Test.Sdk and a runner adapter; dotnet test finds, runs and reports tests.
  • Assertions throw when expectations fail. Put the expected value first and use Throws for exceptions.
  • Theories turn lists of cases into separately reported tests; use member or source attributes for non-constant data.
  • xUnit makes a new test class instance per test (constructor setup, Dispose cleanup) and shares expensive resources through fixtures.
  • Async tests return Task, never async void.
  • You cannot run the real packages in our editor, but a miniature runner with the same attributes shows exactly how they work.

Next: Mocks, Stubs and Fakes