How many integrations with external systems does a modern web-application have? For example, let’s take the checkout process of an online store.

A customer places goods into a cart, and the checkout process starts. Then the customer fills in a shipping address, and the store asks the customer to pay the total order amount and redirects him to the Stripe payment service. When the payment is processed, Stripe redirects the customer back to the store. If the payment was successful, the store presents payment details and sends an email to the customer with SendGrid mail service. Under the hood, the order is sent to Salesforce CRM to be processed by store’s employees. By the end of month, orders are exported to Power BI for analytics.

This is a very simplified example of an online store business process with 4 integrations - Stripe, SendGrid, Salesforce, and Power BI.

A modern e-commerce system can have from 10 to 20 integrations with other systems - for payment processing, notifications, storage, stock tracking and a warehouse, marketing campaigns, analytics, etc.

Nowadays, there is little sense to invent a wheel instead of using existing COTS technologies, available as SaaS.

To keep customers happy, the software should be thoroughly tested to eliminate bugs, which often happen on the integration point between the two systems.

What to test?

The most valuable for the business is the happy day scenario, when a customer picks goods up, pays the total order amount, and gets the order delivered. It definitely should be thoroughly tested since it generates profit for a store.

But it’s important to test exceptional or alternative scenarios to prevent loss and customers’ dissatisfaction.

Some cases have a business-oriented rather than a technical nature, so they’re the expected alternative conditions in the business process workflow. These conditions will happen whether the business process is automated or not. For example, a customer does not have enough money to pay for the order.

Let’s consider an offline store workflow. A cashier tells the customer that his order cost is $99. A customer looks into his wallet, sees that there is only $50 left, and says to the cashier he has not enough money. The cashier will either cancel the order, or suggest the customer to pay with a credit card.

Let’s consider an online workflow. A customer fills in his credit card info, and presses “Pay”. A payment provider tries to charge $99, a bank rejects the payment since there is not enough money, and the payment provider returns an error to the store. Now it’s up to the store to handle this case and explain what happened to the customer.

The store can explain to the customer that there is not enough money on his credit card, and can suggest to try again. Otherwise, the store will automatically cancel the order in 1 day.

Either way, the store should show a meaningful error description to his customer. The customer won’t be happy to see “Internal error occurred” and will have no clue what to do next.

Exceptional cases have a technical nature rather than a business one. Business would prefer not to have these cases happen at all, but nevertheless they should be taken into account. For example, a store has a limit of 1000 transactions per day according to the purchased payment provider’s service plan. The technical personnel had forgotten about this limitation prior to Christmas sales when the limit should have been bumped up. 1001’s customer’s order has not been processed by the payment provider, and API returns an error to the store backend. But the store backend does not know how to deal with this particular error, since it had not been expected by developers. The sad customer sees “Sorry, something went wrong on our side.” and goes to some competitor store.

An order processing should have been tested against both alternative conditions and technical errors of a payment provider.

How to test?

Using a real API

It’s the most straightforward and obvious way. Let’s put aside the e-commerce shop for a moment and look at the CMS integration example. A government system tracks applications of citizens, and uses a CMS to store documents attachments. Each attachment file in the CMS should have metadata filled with document details.

The test requests the document service to create a document, and verifies that the document is created in CMS. Since a good test should not modify global state to keep tests isolated and repeatable, it cleans the created document up.

[Test]
public void Test_LaserficheService_PutDocument_TiffNotCompressed()
{
    LaserficheDocumentService service = InitService();
    IDocument expectedDoc = CreateTestDocument(TestFileType.ImageSmallTiff);

    string documentId = service.PutDocument(expectedDoc);
    IDocument actualDoc = service.GetDocument(documentId);
    service.DeleteDocument(documentId);

    Assert.AreEqual(expectedDoc.DocumentData.Length, actualDoc.DocumentData.Length);
}

Testing real API gives a high level of confidence in the code, but it’s expensive in terms of performance.

It may require a complex setup code to verify alternative scenarios. It makes tests fragile, since if API becomes unavailable then the tests fail.

It has a narrow area of application, in cases when API calls have visible side-effects. We should not charge a credit card or send emails to customers in tests.

Some services provide a sandbox or test credentials for developers. All the actions will be processed in an isolated sandbox, that does not affect the real world. For example, some payment providers have test cards.

To sum up, the testing against real API:

• - Gives high level of confidence in the happy path scenario
• + Makes testing alternative scenarios hard
• + Makes tests fragile and depends on API availability
• + Has a narrow area of application

Using a fake API server

Set up your own server that will mimic real API and return predefined responses. To get responses, work with real API and record responses for various cases and scenarios.

It allows developers to work with API even when API does not yet exist, at all. Discuss the API with another team, vendor or contractor - and get the specification. Set up your own server with a fake API which returns canned responses according to specification. Now you’re able to develop the code using API, while other developers work on actual API implementation.

A fake API server could give you false confidence in the code, if an external API provider introduces breaking changes in behavior. Tests won’t catch that since they use canned responses. Nevertheless, most API providers do not introduce breaking changes for existing API and use versioning to keep backward compatibility. Breaking changes goes against agreement in a defined contract or an API specification.

A fake API server enables you to test even a third-party code you have not control on. To do it, set up your testing environment to redirect requests to real API to fake API server, e.g. using system hosts file:

127.0.0.1:9000 api.payments.com

For .NET we can use WireMock to set up a fake API server.

private FluentMockServer server;
private readonly static string PaymentApiBaseUrl = "http://localhost:9000";

[SetUp]
public void SetUp()
{
    server = FluentMockServer.Start(new FluentMockServerSettings
    {
        Urls = new string[] { PaymentApiBaseUrl }
    });

    server
        .Given(Request.Create()
        .WithPath("/charge")
        .WithBody(JsonConvert.SerializeObject(new
        {
            OrderId = 1,
            Amount = 99.0m,
        }))
        .UsingPost())
        .RespondWith(Response.Create()
        .WithStatusCode(HttpStatusCode.OK)
        .WithHeader("Content-Type", "application/json")
        .WithBodyAsJson(new
        {
            success = true,
            transaction_id = 777,
        }));
}

Happy path scenario test:

[Test]
public async Task ChargeOrder_WhenPaymentProcessed_ShouldMarkOrderAsPaid()
{
    Order order = new Order(id: 1, total: 99.0m);
    WiredHttpOrderService service = new WiredHttpOrderService(PaymentApiBaseUrl);

    await service.ChargeOrder(order);

    Assert.IsTrue(order.IsPaid);
}

Order service process payment as follows:

public async Task ChargeOrder(Order order)
{
    StripePayment payment = await ChargePayment(order.Id, order.Total);
    try
    {
        order.MarkAsPaid(payment.TransactionId);
    }
    catch (StripePaymentException e)
    {
        order.RecordPaymentError(e.Message);
    }
}

Under the hood, the Stripe payment provider’s API is called out:

private async Task ChargePayment(int orderId, decimal amount)
{
    using (HttpClient httpClient = new HttpClient())
    {
        StringContent body = new StringContent(
        JsonConvert.SerializeObject(new { OrderId = orderId, Amount = amount }),
        Encoding.UTF8,
        "application/json");

        HttpResponseMessage httpResponse = await httpClient.PostAsync($"{paymentApiBaseUrl}/charge", body);
        JObject response = JObject.Parse(await httpResponse.Content.ReadAsStringAsync());

        if (!httpResponse.IsSuccessStatusCode)
        {
            throw new StripePaymentException(response["error"].ToString());
        }

        return response.ToObject();
    }
}

To sum up, a fake API server:

• + eliminates dependency on external API limitations like an amount of requests per day
• + eliminates dependency on external API availability
• + gives high level of confidence in catching HTTP-related issues since it’s executed during a test
• + allows developing even without an existing API, when you have request/response specification defined
• + allow testing a third-party code with requests redirection to a fake API server
• - requires an additional setup
• - can give you a false confidence in your code when an external API provider introduces breaking changes

Using a fake HTTP client

Could we do testing without hosting a fake API server? It’s a significant complication of the tests and the environment setup.

Sure, move a test seam to the level of a HTTP client used by a system under the test. Then, set up a HTTP client to return fake responses without actually sending a request over the wire to the API server.

In order to go with this approach, you should have control on the HTTP client used by the code. If it’s a third-party code like SDK designed with testability in mind, it could provide a way to inject your own HTTP client implementation for testing.

With a fake HTTP client, it’s easy to test various error conditions like service unavailability, rate limiting or SSL error.

Let’s look at the test:

[Test]
public async Task ChargeOrder_WhenCustomerHasNotEnoughMoney_ShouldRecordLastPaymentError()
{
    Order order = new Order(id: 1, total: 99.0m);

    HttpClient fakeHttpClient = new HttpClient(
        FakeHttpResponse(
            new HttpRequestMessage
            {
                Method = HttpMethod.Post,
                RequestUri = new Uri($"{PaymentApiBaseUrl}/charge")
            },
            new HttpResponseMessage
            {
                StatusCode = HttpStatusCode.Conflict,
                Content = new StringContent(JsonConvert.SerializeObject(new
                {
                    success = false,
                    error = "Insufficient balance",
                }))
            }));

    InjectedHttpOrderService service = new InjectedHttpOrderService(PaymentApiBaseUrl, fakeHttpClient);

    await service.ChargeOrder(order);

    Assert.AreEqual("Insufficient balance", order.LastPaymentError);
}

In .NET, you can use HttpMessageHandler to set up predefined responses and create HttpClient that use your HttpMessageHandler HttpClient  that use your HttpMessageHandler

private static HttpMessageHandler FakeHttpResponse(HttpRequestMessage request, HttpResponseMessage response)
{
    return new FakeHttpMessageHandler(request, response);
}

private class FakeHttpMessageHandler : HttpMessageHandler
{
    private readonly HttpRequestMessage request;
    private readonly HttpResponseMessage response;

    public FakeHttpMessageHandler(HttpRequestMessage request, HttpResponseMessage response)
    {
        this.request = request;
        this.response = response;
    }

    protected override Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        bool requestMatched = this.request.Method == request.Method && 
            this.request.RequestUri == request.RequestUri;

        if (!requestMatched)
        {
            throw new Exception($"Unexpected request: {request.RequestUri}");
        }

        return Task.FromResult(response);
    }
}

Introduce a seam to inject HttpClient implementation and use it instead of the directly instantiated one.

public InjectedHttpOrderService(string paymentApiBaseUrl, HttpClient httpClient)
{
    this.paymentApiBaseUrl = paymentApiBaseUrl;
    this.httpClient = httpClient;
}

To sum up, a fake HTTP client:

• + removes overhead of having a fake API server
• + enables testing HTTP-related errors like SSL handshake fault
• - requires ability to inject a fake HTTP client implementation

Using a fake API client

Get rid of HTTP-related details like API endpoint URLs and error codes by introducing abstraction over interaction with API via HTTP - API client. API client interface should reflect operations defined in API specification, like “charge payment”, “get payment status”, etc.

It will hide implementation details of sending requests, handling response codes and deserialization.

All the possible errors that API returns should be included into an API client specification. It could be either an exception or model returned with operation status and error codes.

This allows faking API client behavior according to scenarios under test.

[Test]
public async Task ChargeOrder_WhenPaymentIsProcessed_ShouldMarkOrderAsPaid()
{
    Order order = new Order(id: 1, total: 99.0m);

    IStripePaymentApiClient fakePaymentApiClient = Substitute.For();
    fakePaymentApiClient.ChargePayment(orderId: order.Id, amount: order.Total).Returns(new StripePayment
    {
        TransactionId = "777"
    });

    StripePaymentApiOrderService service = new StripePaymentApiOrderService(fakePaymentApiClient);

    await service.ChargeOrder(order);

    Assert.IsTrue(order.IsPaid);
}

An order service has a testing seam and allows an API client injection in constructor

public StripePaymentApiOrderService(IStripePaymentApiClient paymentApiClient)
{
    this.paymentApiClient = paymentApiClient;
}

public async Task ChargeOrder(Order order)
{
    try
    {
        StripePayment payment = await paymentApiClient.ChargePayment(order.Id, order.Total);
        order.MarkAsPaid(payment.TransactionId);
    }
    catch (StripePaymentException e)
    {
        order.RecordPaymentError(e.Message);
    }
}

What are the downsides of the fake API client approach? To use it, a developer should have a very clear understanding of how API behaves in different conditions to define an accurate interface for an API client. An API client interface is a kind of idealized perception, approximation of real API.

Logic is tested with a fake API client which does not call out real API. When the same logic executes in production, it will use a live API client implementation.The implementation could have bugs in it, API could behave in unexpected ways under circumstances, unforeseen during the development. This will break logic, in spite of green tests that say everything is fine.

To minimize risks, the production API client implementation should be tested, too. It will be a different set of tests, to ensure that the production API client implementation actually conforms to the specification of its interface. For example, when API returns error code 429 “Too Many Requests”, the API client should throw TooManyRequestsException which is expected by the code relying on the API client.

All the bugs related to lack of knowledge of real API behavior should be included into the API client test suite, and API client interface should be updated to accommodate this revealed knowledge.

For many services, an API client could exist in the form of SDK.

To sum up, a fake API client:

• + Hides HTTP-related details and moves developer’s focus towards logic of domain-specific operations
• + Simplifies testing of alternative scenarios
• - Requires very accurate understanding of the real API behavior
• - May give you a false confidence in the logic, due to missed details of real API behavior

Use service (Anti-Corruption Layer)

The API client approach has one significant drawback - it tightly couples business logic to the concrete payment provider, which could lead to a vendor’s lock-in. Payment implementation details seep through the codebase and make it hard to change a payment provider. Exceptions and models specific to the concrete payment provider are spread all over the code as time passes.

One would say:

• Why should we ever care about changing a payment provider? It’s an unnecessary over-complication!

But the market changes, as long as business requirements. A payment provider might experience financial troubles and go bankrupt. A new payment provider might quickly gain their market share with attractive terms of service and lower commission fees.

Given that, one day a business would say that they’d like to use a new payment provider to pay less fees, and it’s up to developers to accommodate this change.

To overcome a vendor’s lock-in, introduce an Anti-Corruption Layer (ACL) which will isolate the business logic from third-party payment processing details. The Anti-Corruption Layer should provide a unified interface for payments processing.

ACL introduces a clear boundary between business logic and third-party service. Application logic is built upon business operations like “charge payment”, “refund payment” and could be tested separately from a concrete payment provider. A payment provider could be switched with introducing another payment gateway implementation.

To build a good ACL, we should define an interaction protocol for the external service.

How to define interaction protocol for ACL?

Identify the main operations with their alternative/exceptional conditions for the business process.

The operation “charge the payment for order” becomes the Charge method on the payment gateway. Alternative condition “there is not enough money to pay” becomes an exception thrown out of the Charge method.

How to isolate concrete service implementation details from business logic operations?

Ask yourself: if we change the service provider, will this detail disappear?

We live in a real world and know that any abstraction could leak. A payment provider could become temporarily unavailable, like a bank could be closed for the weekend in an offline world. Changing the payment provider won’t eliminate this case, so it should be included into the interaction protocol, let it be the ServiceTemporaryUnavailable exception.

Most payment providers have limits for the number of requests per second, so called rate limiting. Switching the payment provider likely won’t save us from the imposed limitations, so it should be included into the protocol, too. Let it be the RetryLaterException.

Let’s assume we know from the practice that PayPal sporadically returns Unauthorized error, and from the investigation showing that a retry will overcome this issue in 9 of 10 cases. Changing PayPal to Stripe could eliminate this problem, so it’s PayPal implementation detail and should not be included into a payment processing interaction protocol.

Consider another case when a customer does not have enough money to pay - this is a part of protocol, since this case hardly depends on a payment provider and will happen for sure.

Testing business logic against ACL

public async Task ChargeOrder_WhenCustomerHasNotEnoughMoney_ShouldRecordLastPaymentError()
{
    Order order = new Order(id: 1, total: 99.0m);

    IPaymentGateway fakePaymentGateway = Substitute.For();
    fakePaymentGateway.ChargeOrder(order).Throws();

    PaymentGatewayOrderService service = new PaymentGatewayOrderService(fakePaymentGateway);

    await service.ChargeOrder(order);

    Assert.AreEqual("Not enough money", order.LastPaymentError);
}

Order service implementation:

public PaymentGatewayOrderService(IPaymentGateway paymentGateway)
{
    this.paymentGateway = paymentGateway;
}

public async Task ChargeOrder(Order order)
{
    try
    {
        Payment payment = await paymentGateway.ChargeOrder(order);
        order.MarkAsPaid(payment.ExternalPaymentId);
    }
    catch (NotEnoughMoneyException e)
    {
        order.RecordPaymentError("Not enough money");
    }
    catch (PaymentException e)
    {
        order.RecordPaymentError(e.Message);
    }
}

Payment gateway implementation for Stripe

public StripePaymentGateway(IStripePaymentApiClient apiClient)
{
    this.apiClient = apiClient;
}

public async Task ChargeOrder(Order order)
{
    try
    {
        StripePayment payment = await apiClient.ChargePayment(order.Id, order.Total);
        return MapToPayment(payment);
    }
    catch (StripeInsufficientFundsException e)
    {
        throw new NotEnoughMoneyException("Not enough money", e);
    }
    catch (StripePaymentException e)
    {
        throw new PaymentException("Sorry, payment failed", e);
    }
}

private Payment MapToPayment(StripePayment payment)
{
    return new Payment(payment.TransactionId);
}

Payment gateway implementation for PayPal

public PaypalPaymentGateway(IPaypalApiClient apiClient)
{
    this.apiClient = apiClient;
}

PayPal provides SDK with the following interface:

public interface IPaypalApiClient
{
    Task Pay(int orderId, decimal amount);
}

Instead of throwing an exception, SDK returns operation status with error code:

public class PaypalOperationStatus
{
    public string PaymentId { get; set; }
    public PaypalErrorCode ErrorCode { get; set; }
}

Anti-Corruption Layer isolates nitty-gritty details of PayPal SDK, and transforms SDK interface to adhere application payment gateway protocol.

public async Task ChargeOrder(Order order)
{
    PaypalOperationStatus paymentOperationStatus = await apiClient.Pay(order.Id, order.Total);
    if (paymentOperationStatus.PaymentId != null)
    {
        return MapToPayment(paymentOperationStatus);
    }
    else if (paymentOperationStatus.ErrorCode == PaypalErrorCode.InsufficientBalance)
    {
        throw new NotEnoughMoneyException("Sorry, not enough money");
    }
    else
    {
        throw new PaymentException($"Payment failed. Internal error: {paymentOperationStatus.ErrorCode}");
    }
}

private Payment MapToPayment(PaypalOperationStatus status)
{
    return new Payment(status.PaymentId);
}

Handling service-specific failures

We remember that PayPal sporadically returns Unauthorized error, and a retry helps to overcome it. This nitty-gritty detail should be definitely handled with the code to make it more resilient. A payment gateway for PayPal is the perfect place to deal with this error.

Let’s implement the test for this scenario:

[Test]
public async Task Charge_WhenOperationIsNotAuthorized_ShouldRetryPayment()
{
    string paymentId = "777";
    Order order = new Order(id: 1, total: 99);

    IPaypalApiClient fakeApiClient = Substitute.For();
    fakeApiClient.Pay(Arg.Any(), Arg.Any()).Returns(
        new PaypalOperationStatus { ErrorCode = PaypalErrorCode.UnauthorizedOperation },
        new PaypalOperationStatus { PaymentId = paymentId });
                
    PaypalPaymentGateway paymentGateway = new PaypalPaymentGateway(fakeApiClient);
    Payment payment = await paymentGateway.ChargeOrder(order);
    Assert.AreEqual(paymentId, payment.ExternalPaymentId);
}

And implement retries using Polly library:

public async Task ChargeOrder(Order order)
{
    PaypalOperationStatus paymentOperationStatus = await Policy.HandleResult(
        status => status.ErrorCode == PaypalErrorCode.UnauthorizedOperation)
    .RetryAsync(10)
    .ExecuteAsync(async () => await apiClient.Pay(order.Id, order.Total));
    
    if (paymentOperationStatus.PaymentId != null)
    {
        return MapToPayment(paymentOperationStatus);
    }
    else if (paymentOperationStatus.ErrorCode == PaypalErrorCode.InsufficientBalance)
    {
        throw new NotEnoughMoneyException("Sorry, not enough money");
    }
    else
    {
        throw new PaymentException($"Payment failed. Internal error: {paymentOperationStatus.ErrorCode}");
    }
}

To sum up, ACL:

• + enables testing of business logic independent of API interaction details
• + enables swapping service provider without changing business logic
• + enables handling service-specific problems separately from business logic code
• - requires preliminary work to define interaction protocol
• - adds implementation overhead and makes development longer

Conclusion

We went through different approaches to testing interaction with external services. There is no one-fits-all, right or the best method. It will always depend on the project specific requirements, development team and their skills, and a delivery schedule. In my opinion, the API client approach is a good enough compromise between complexity and flexibility that enables testing logic without fiddling around much.

Demo project is available on GitHub