Different kind of testing strategies with ASP.NET Core: The basics

Most enterprises or development teams will have a quality gate in place, which dictates a percentage of code coverage. When writing new or changing your existing code, you also need to supplement them with unit tests. I enjoy writing unit tests when they have a more functional purpose. For example, creating orders containing products with discounts, testing a regression, or confirming the cause of a bug. Looking back at some of the tests I've been writing myself with ASP.NET Core, there is more than meets the eye. These tests fit more in a different category of the testing pyramid. From top to bottom, we have:

End-to-End tests
UI tests
Integration tests
Unit tests

The higher we go in the pyramid, the more effort it takes to write these tests. During a "rebuild" project, we had a dedicated tester available. This rebuild project was to migrate some REST API's from ASP.NET 4.5 into ASP.NET Core. With containerization being a trend, we also decided to deploy and develop these API's using Docker. As we were optimizing our workflow, we also needed some initial end-to-end testing to be done. However, the work done by our tester was mostly manual (both for the legacy and the new applications).

While we were producing both ends of the spectrum as a team, we were still looking to increase our throughput for deliverables. When we started automating part of our test suite, we stumbled into the component/integration test area. But, our tester, who could write code, wasn't very well-known in the .NET core/Docker space. So we helped him with writing some code to self-host the app to send HTTP requests to the API.

Our example application
For these blog posts, I will use an example application to explain the ideas behind component testing. I build a small REST API application with dependencies to a SQL Database and an integration with an HTTP API. At the bottom of this blog, you will find the code in the resources.

What's next?
In the next couple of blog posts, I will describe how we can get started with ASP.NET Core and xUnit to create a component or an integration test. As a foundation for the other posts, this first blog post teaches you:

The basics of self-hosting the ASP.NET Core application in xUnit
Using WebApplictionFactory and how to customize the startup
Change/configure settings for your test
How to replace dependencies such as an SQL database, RabbitMQ or HTTP clients
View everything logged by your application during a test

With this introduction out of the way, let us get started.

| Note: While each test in the pyramid has a different goal, xUnit doesn't classify or mark these tests differently. A test in xUnit is a method which means that a method can be a unit test, a component test, or an integration test.

Sending HTTP requests to your ASP.NET Core application in a unit test.

In our example application, we have a REST API. It is written with ASP.NET Core, and to start sending HTTP requests to this application, we need a way to host it. To achieve this, we need to carry out a couple of steps. ASP.NET Core was built with extension in mind and comes out of the box with self-hosting capabilities. In the Microsoft.AspNetCore.Mvc.Testing NuGet package, you will find a WebApplicationFactory<TEntryPoint> class (where TEntryPoint is either your Startup or Program). This class is capable of self-hosting the application anywhere in your code. Once this factory has started our API, we can then use this factory to resolve a HttpClient.

We can choose to create or inject our factory and resolve the HttpClient. This factory will do a couple of things for us. It establishes a TestServer that runs our self-hosted application. This server is responsible for hosting the application. It is similar to IIS, Kestrel or whatever your flavour might be. It configures the entire request pipeline and then starts processing requests. With the HttpClient, we then send out an HTTP request to an endpoint; in our case, this is GET /api/v1/orders, and we assert that the response is 200 OK.

public class OrderApiComponentTest
{
    private readonly WebApplicationFactory<Program> _webApplicationFactory;

    public OrderApiComponentTest()
    {
        _webApplicationFactory = new WebApplicationFactory<Program>();
    }

    [Fact]
    public async Task When_GET_is_called_on_the_api_should_return_200_OK()
    {
        // Arrange

        // Act
        HttpResponseMessage response = await SystemUnderTest.GetAsync("/api/v1/orders");

        // Assert
        response.StatusCode.Should().Be(HttpStatusCode.OK);
    }

    public HttpClient SystemUnderTest =>
        _webApplicationFactory.CreateClient();
}

With this as our foundation, we can start looking into more details. Depending on your application, this test might fail due to dependencies such as databases or other services not connecting. But we will try to resolve this along the way.

WebApplicationFactory<TEntryPoint> deep dive

Startup or Program
The WebApplicationFactory<TEntryPoint> is used to create a TestServer. The TEntryPoint is a class that can either be the Startup or the Program. It doesn't really matter which one you use. The factory will use to entry point class use the assembly of the class to then try to discover one of the following static methods:

IWebHost CreateWebHost(string[] args)
IWebHostBuilder CreateWebHostBuilder(string[] args)
IHostBuilder CreateHostBuilder(string[] args)

When discovered, this method is then invoked and used to start and run the app. If it doesn't find this public static method, it will fail to startup.

Some NuGet packages or libraries that are available will tell you to customize this builder/host method. For them to hook into the ASP.NET Core runtime. For example, AutoFac is a library that changes the ServiceProvider. And we have written some libraries for our own projects that used custom initialization code in our Program Main() method to perform additional initialization tasks.

However, the factory doesn't invoke this Main() method. In the below (incorrect) example, the Main() method declares that AutoFac should be used as ServiceProvider. But because the factory only invokes the static method, AutoFac will not be configured for our test. So if you decide to add additional startup code to your main method, make sure it is invoked separately in your test. Or, you can create a custom class that inherits WebApplicationFactory<T> that does this for you.

public class Program
{
    // Any custom code here is not invoked by WebApplicationFactory.
    public static async Task Main(string[] args)
    {
        IHostBuilder hostBuilder = CreateHostBuilder(args);
        hostBuilder.UseServiceProviderFactory(new AutofacServiceProviderFactory())
        IHost host = hostBuilder.Build();
        await host.RunAsync();
    }

    // Discovered and invoked by the WebApplicationFactory
    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)            
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            });
}

Resolving services to arrange your data
The WebApplicationFactory not only gives us the ability to host the application, but it also exposes the IServiceProvider of our application. This means we can resolve all of our services that have been registered inside our application. For example, if we need to pre-populate our database with some customers, we can resolve a repository to add some data.

If your classes are registered as singleton or transient, you can use the Services property on the factory. But if your classes are registered as scoped, you also need to resolve these services as scoped. We will populate our customers in the arrange step of our test.

// Arrange
IRepository<Customer> customers = 
    _webApplicationFactory.Services.GetRequiredService<IRepository<Customer>>();
customers.CreateAsync(new Customer(...));

// Arrange in scoped
using (IServiceScope scope = _webApplicationFactory.Services.CreateScope())
{
    IRepository<Customer> customers = scope.ServiceProvider.GetRequiredService<IRepository<Customer>>();
    customers.CreateAsync(new Customer(...));
}

Adding custom appsetting.json

Even Though we are self-hosting our application in a xUnit test, the application works the same as when you host in on IIS, cloud or Docker environment. One of the things ASP.NET Core automatically does is load a bunch of settings from different sources. But for your tests, you might need a way to override these. The runtime loads them in the following order: app settings, environment-specific settings, user secrets, system environment, and last but not least, the console arguments.

To override these settings, we can customize the factory to customize our web host. Our goal is to load additional app configurations specific to only our test. In the factory class, we have to call the WithWebHostBuilder method, which is a callback with the IWebHostBuilder as parameter.

public class OrderApiComponentTest
{
    private readonly WebApplicationFactory<Program> _webApplicationFactory;
    private readonly ITestOutputHelper _testOutputHelper;

    public OrderApiComponentTest(ITestOutputHelper testOutputHelper)
    {
        _webApplicationFactory = new WebApplicationFactory<Program>().WithWebHostBuilder(CustomizeWebHostBuilder);
        _testOutputHelper = testOutputHelper;
    }

    private void CustomizeWebHostBuilder(IWebHostBuilder webHostBuilder)
    {
        webHostBuilder
            .ConfigureAppConfiguration((context, configuration) =>
            {
                string folder = Path.GetDirectoryName(GetType().Assembly.Location);
				string file = "ComponentTest/component.test.json";
				configuration.AddJsonFile(Path.Combine(folder, file));
            });
    }
}

To load the test-specific settings, you need to make sure you fully specify the path to your settings file. The application has a content root and a webroot. The content root is the one that is used to load assemblies, JSON/XML/DB files, and Razor views. But this content root is located in the project that you are self-hosting. In our case, that is the "Api" project. So when you only specify, for example, "ComponentTest/component.test.json", it will not find the file because it is located in our "Api.Test" project.

So with the customizations in place for the web host, we can edit our JSON file to include settings. Additionally, we can write a test to assert the correct settings are loaded instead of our original values from the user secrets.

{
  "Shopify": {
    "ApiKey": "COMPONENT_TEST"
  }
}

[Fact]
public async Task When_application_starts_up_should_load_config_from_json_file()
{
    // Arrange
    IConfiguration configuration = _webApplicationFactory.Services.GetRequiredService<IConfiguration>();

    // Act/Assert
    configuration["Shopify:ApiKey"].Should().Be("COMPONENT_TEST");
}

Changing dependencies by using Test specific Dependencies

If you use a relational database, a message queue or a NoSql database, we need to start thinking about how these dependencies will work if we self-host the application. During the development of your app or executing these tests locally on your own machine, they might have a chance of succeeding. But this might not be the case for everything. And if you have a continuous integration (CI) pipeline, running these tests on that build agent could also fail. The goal of these component/integration tests is that we are trying to make them so that they can also run inside a CI/CD environment.

So we need to replace some of our dependencies. Generally speaking, a Startup class will have a method called ConfigureServices(IServiceCollection services) where all our dependencies are registered. But during our tests, we might need to replace some services to pass the tests. For example a IRepository, IBasicConsumer or any other dependency with an abstraction. For now, we will be replacing these services with a TestDouble.

To achieve this, we will call a special method ConfigureTestServices on the host builder. This is a special callback that is executed after your Startup.ConfigureServices and is only available when used in combination with the WebApplicationFactory<T>. This method allows you to override any previously registered services and replace them with something else, like a fake. Alternatively, this callback is also a place to configure test specific behaviour further. For example, configuring EntityFramework to use the SQLite implementation instead of a SQL server or, change how HTTP client/services are resolved.

private void CustomizeWebHostBuilder(IWebHostBuilder webHostBuilder)
{
    webHostBuilder
        .ConfigureTestServices(services =>
        {
            services.AddScoped<IRepository<Customer>, InMemoryRepository<Customer>>()
            services.AddSingleton<IBasicConsumer>(new FakeBasicConsumer())
                    
            // Etc
        });
}

Adding output using logging with xUnit

Last but not least, we are going to take a look at logging. Again ASP.NET Core automatically sets this part up for us. But in your company, you might be using an Application Performance Management (APM) tool such as DataDog or Application Insights. If you have configured these third party APM systems, your component or integration test will be sending logs to these tools. So we want to prevent that, but when debugging or investigating issues, it is still useful to have some form of logging. xUnit comes out of the box with a ITestOutputHelper class. Anything that is written to that class during your test is stored as an output with the test.

Together with the NuGet package, Xunit.Extensions.Logging we can register this class as an implementation of ILogger. First, we have to remove all of the other log providers to send anything to your APM. And then, we will add the xUnit ITestOutputHelper class. Any logs will then be written as output and be available in the test explorer, as shown in the above picture. This must also be done to customize our web host builder by using the ConfigureLogging() method. First, we accept ITestOutputHelper as a constructor parameter of the test class. And then, we register the xUnit class with our logging builder.

// Accept ITestOutputHelper as constructor parameter.
public OrderApiComponentTest(ITestOutputHelper testOutputHelper)
{
    _webApplicationFactory = new WebApplicationFactory<Program>().WithWebHostBuilder(CustomizeWebHostBuilder);
    _testOutputHelper = testOutputHelper;
}

private void CustomizeWebHostBuilder(IWebHostBuilder webHostBuilder)
{
    webHostBuilder
        .ConfigureLogging(loggingBuilder =>
        {
            loggingBuilder.ClearProviders();
            loggingBuilder.AddXunit(_testOutputHelper);
        });
}

Conclusion

With this post as our foundation, we will continue to explore writing component and integration tests for ASP.NET Core. You might have already seen some hints of what a "component test" can look like. This post has shown you some of the basics that we will be using when testing an ASP.NET Core application in a xUnit test. We have gone through:

Self-hosting our web application in a test using xUnit
Customizing the application with WebApplicationFactory
Adding test specific settings
Replacing dependencies specific to the test
Replacing existing logging with the xUnit output for additional debugging purposes

We will go deeper into how we can write our component and integration tests in the next posts.