Painless contract testing with Pactflow

Updates (September 2022)

Pactflow updated the Github actions, so the publish-provider-contract action since version 1.0.0 can now publish the contract even if tests fail, and the can-i-deploy action from version 1.0.0 now accepts a parameter for “–to-environment”, and in our demo consumer we now also use the Pactflow action to publish our pact files which needs less parameters.

Following up on the contract testing with Pact series, it is now a good time to look at one of the features of Pactflow, the SaaS platform for simplifying contract testing: Bi-directional contract testing.

If you are not familiar with the concept of Pact, for your peace of mind, I would highly recommend reading up on our previous articles on the topic. Especially the post about writing Pact contract tests in Spring Boot for better comparison. And if you already know it, even better. Let’s dive right in!

Why bi-directional?

While consumer driven contract testing works great, you might have experienced some challenges during use. The initial learning curve is pretty steep, which can then result in negligence of Pact tests and slow adoption. Few projects actually manage a “consumer first” approach, so in some cases teams struggle with adoption of feature changes due to consumer requests. In other cases there are many consumers with differing needs and different test cases, and the provider team is busier with adjusting verification tests than actually providing new features. Adding consumer driven Pact tests onto existing systems at a later point is quite the effort.

Bi-directional contract testing in contrast, supports an API first approach, is easier to retrofit and you can also apply it to testing any external API. It increases the independence between consumer and provider application development immensely, and has a much lower adoption barrier.

In consumer driven Pact testing, you as the consumer of the API would implement tests to verify the interaction with the API against a Pact Mock Provider to assure that the response for a given request matches your expectations. Usually you define the provider states in close collaboration with the API provider team. When your tests are successful, you upload the generated contracts, or in this case “pacts” to the Pact broker. The provider team then has to implement the contract verifications for each state that matches the expected outcome defined by the consumer.

In bi-directional contract testing the dependence is a lot more relaxed. Both teams can write their own tests based on the API specification independently. The Pactflow broker uses the API specification file to check compatibility of both sides without the provider needing to accommodate the consumer with specific states.

In this blog post we will present the main changes to your testing approach when using Pactflow bi-directional contract testing instead of consumer-driven Pact. As a sample scenario, we will use two Spring Boot microservices for employees management, a provider and a consumer, respectively. In particular, we will show the smoother approach for verifying the provider API compatibility, while keeping consumer-driven Pacts on the consumer side.

Project setup

Our sample microservices are Spring Boot applications, and you can find the sources for the consumer and provider on GitHub. The provider exposes REST endpoints with a GET and POST method.

As a CI tool for building our project and running our contract tests, we will use GitHub Actions, but you are free to pick the CI tool of your choice.

For the full experience of the examples, you will need a Pactflow trial to set up your own broker. Don’t worry, you won’t need to provide any billing data, and you can run up to five contracts with the free trial. Once you have set that up, you only need to replace the broker URL and secret tokens.

The provider project

First we implement our controller and our request and response objects. If you go spec first, you could even have the objects created automatically based on your specification.

Additionally we need to provide the Pact broker with the Open API definition file for comparing compatibility.

In the sample project, this is done during the test step of the build pipeline:

Snippet from main.yml

- name: Build with Maven
  run: mvn clean verify -B
- name: Publish provider contract on passing test run  
  if: success()
  uses: pactflow/actions/publish-provider-contract@v1.0.0
  env:
    oas_file: src/main/resources/openApi/openapi.yml
    results_file: target/surefire-reports/de.kreuzwerker.blogs.bidirectionalprovider.DemoControllerTest.txt
- name: Publish provider contract on failing test run
  # ensure we publish results even if the tests fail  
  if: failure()
  uses: pactflow/actions/publish-provider-contract@v1.0.0
  env:
    oas_file: src/main/resources/openApi/openapi.yml
    results_file: target/surefire-reports/de.kreuzwerker.blogs.bidirectionalprovider.DemoControllerTest.txt
    # ensure we set the EXIT_CODE to ensure we upload a failing self-verification result
    EXIT_CODE: 1

Here we use this comfy GitHub Action provided by Pactflow, where we need to add the oas_file which represents the path to the specification file (can be json or yaml) and the results_file. The results_file represents the report of your own tests. Since there is no longer a contract verification, you yourself are responsible for ensuring your compatibility to the specification, so this file basically represents the proof.

Find the list of all parameters for the action here or read up on the Pactflow documentation of the publish provider contract command. So this action takes care of defaulting the content type of the file to text/plain, uses the current branch and generates a verifier tool ‘github-actions’ if you do not provide the parameter. The verifier tool is currently not used, but might get more interesting in the long run and maybe even be displayed in the Pactbroker UI.

And that’s it for additional pipeline configuration. (No more extra trigger for a verification step!)

Instead of a contract verification test, we can now write our controller tests and simply add the OpenApiValidationFilter by Atlassian to ensure that our requests and outgoing responses match what we defined in our Open API spec file.

Snippet from de.kreuzwerker.blogs.bidirectionalprovider.DemoControllerTest

private final String specPath = "src/main/resources/openApi/openapi.yml";
private final OpenApiValidationFilter validationFilter = new OpenApiValidationFilter(specPath);

In the example, the spec path is a file path, you can as well use a URL.

Snippet from de.kreuzwerker.blogs.bidirectionalprovider.DemoControllerTest

@Test
void shouldCreateEmployee() throws JSONException {

  JSONObject requestObject = new JSONObject();
  requestObject.put("email", "email@address.com");
  requestObject.put("firstName", "Jane");
  requestObject.put("lastName", "Doe");

  Employee result =
      given()
          .port(randomServerPort)
          .filter(validationFilter)
          .when()
          .request()
          .contentType(ContentType.JSON)
          .body(requestObject.toString())
          .post("/demo-service/v1/departments/{departmentId}/employees", UUID.randomUUID())
          .then()
          .assertThat()
          .statusCode(201)
          .extract()
          .body()
          .as(Employee.class);

  assertThat(result.getEmail()).isEqualTo("email@address.com");
}

The test methods do not really change, the only addition is the RestAssured filter that we pass our validation filter into.

Let’s look at some verification examples by the OpenApiValidationFilter.

DemoControllerTest

@Test
void shouldFailDueToPath() {

  given()
      .port(randomServerPort)
      .filter(validationFilter)
      .when()
      .get("/demo-service/v1/department/{departmentId}/employees", UUID.randomUUID())
      .then()
      .assertThat()
      .statusCode(200);
}

Running this test method, will result in a failure with

com.atlassian.oai.validator.restassured.OpenApiValidationFilter$OpenApiValidationException: {
  "messages" : [ {
    "key" : "validation.request.path.missing",
    "level" : "ERROR",
    "message" : "No API path found that matches request '/demo-service/v1/department/05627e0d-f05d-4191-84e9-7a5f1de70667/employees'.",
    "context" : {
      "requestPath" : "/demo-service/v1/department/05627e0d-f05d-4191-84e9-7a5f1de70667/employees",
      "requestMethod" : "GET"
    }
  } ]
}

In this case the OpenApiValidationFilter complains - correctly - about a request path that is not defined.

Using the OpenApiValidationFilter in its full scope, we can validate the request path, the request object and response object.

Unfortunately, it is currently not possible (or I simply did not find a way) to define several response bodies for one request. For example the standard response for 200 response, and an error object for a 4xx response. So if you want to add your error response tests, and still use the OpenApiValidation filter to take care of at least the request path and body, you need to slightly adapt the use of the filter.

Customized OpenApiValidationFilter

OpenApiInteractionValidator validator =
    OpenApiInteractionValidator.createFor(specPath)
        .withLevelResolver(
            LevelResolver.create()
                .withLevel("validation.request", Level.ERROR)
                .withLevel("validation.schema.required", Level.IGNORE)
                .withLevel("validation.response.body.missing", Level.IGNORE)
                .withLevel("validation.response.body.schema.additionalProperties", Level.IGNORE)
                .withLevel("validation.response.body.schema.required", Level.IGNORE)
                .withLevel("validation.response", Level.IGNORE)
                .build())
        .build();

The validator allows granular control over validation, in this case we want to validate the request, and the response in general, but ignore the body validation.

The consumer project

The consumer project will use both the POST and GET methods from the provider and we start implementing based on the OpenAPI specification file from the provider.

The consumer project implemented their client behavior and set up their tests as “classic” consumer Pact tests.

Snippet from de.kreuzwerker.blogs.bidirectionalconsumer.UpstreamPactTest

@Pact(consumer = "pact-consumer")
public RequestResponsePact pactEmployeeListing(PactDslWithProvider builder) {

  DslPart responseBody =
      newJsonBody(
              (body) -> {
                body.minArrayLike(
                    "employees",
                    1,
                    emp -> {
                      emp.stringType("firstName", "Ellen");
                      emp.stringType("lastName", "Ripley");
                      emp.stringType("email", "ripley@weyland-yutani.com");
                      emp.uuid("employeeId", UUID.randomUUID());
                    });
              })
          .build();

  return builder
      .given("employees exist for department")
      .uponReceiving("a request to list all employees for department")
      .path("/demo-service/v1/departments/" + departmentId + "/employees")
      .method("GET")
      .headers("Accept", MediaType.APPLICATION_JSON_VALUE)
      .willRespondWith()
      .status(200)
      .body(responseBody)
      .toPact();
}

@Test
@PactTestFor(pactMethod = "pactEmployeeListing", pactVersion = PactSpecVersion.V3)
void shouldGetEmployees() {
  EmployeesListing resultListing = demoClient.getEmployees(departmentId).getBody();
  assertThat(resultListing).isNotNull();
  assertThat(resultListing.getEmployees().size()).isGreaterThan(0);
}

Let’s start with a happy path test to ensure that we actually get a list of employees back. And since the spec states that a 404 is returned in case the department Id is not found, let’s check that too and create a quick test for posting a new employee.

Our CI will run the tests and upload the Pact files to the broker.

And the Pactflow UI shows that we are compatible with all our Pacts.

And what about the not-quite-so-happy paths?

Let’s assume the developer was in a rush and they did not read the spec that closely and implemented their Employee object, and wanted to write a test for creating one with only the minimum required data.

Snippet from de.kreuzwerker.blogs.bidirectionalconsumer.UpstreamPactTest

@Pact(consumer = "pact-consumer")
public RequestResponsePact pactEmployeeCreateMinimalData(PactDslWithProvider builder)
    throws JsonProcessingException {
  DslPart responseBody =
      newJsonBody(
              (body) -> {
                body.stringType("firstName", "Ellen");
                body.stringType("email", "ripley@weyland-yutani.com");
                body.uuid("employeeId", UUID.randomUUID());
              })
          .build();

  return builder
      .given("the department exists")
      .uponReceiving("a request to create a new employee with minimal required data")
      .path("/demo-service/v1/departments/" + departmentId + "/employees")
      .method("POST")
      .body(new JSONObject(mapper.writeValueAsString(createEmployeeMinData())))
      .willRespondWith()
      .body(responseBody)
      .status(201)
      .toPact();
}

@Test
@PactTestFor(pactMethod = "pactEmployeeCreateMinimalData", pactVersion = PactSpecVersion.V3)
void shouldCreateEmployeeMinimal() {
  Employee result = demoClient.postEmployee(departmentId, createEmployeeMinData()).getBody();
  assertThat(result).isNotNull();
}

private Employee createEmployeeMinData() {
  Employee emp = new Employee();
  emp.setFirstName("Michelle");
  emp.setEmail("michelle.yeoh@goat.com");
  return emp;
}

In this case the minimum data consists only of first name and email address, whereas the spec file states that last name and email address are required.

Locally the Pact test will run without problems, but on the broker the compatibility tests will fail and complain about the missing property in the request body.

Conclusion

This is one example of how to implement a validation for your both your consumer and provider against the Open API spec file. No more contract verification, no defining of states, no triggering the verification in the provider project after you update a Pact in one of the consumers… Please keep in mind that these contract tests, just like the consumer driven Pact tests, do not replace your functional tests 😉. They validate the API contract only. You still need your own unit and integration tests.

But if you were considering contract testing anyway in addition to your other tests, take a closer look at bi-directional contract testing, especially if you have APIs with many consumers, or work API first anyway. Compared to consumer driven contract testing, you gain more independence, you can add these to existing code easier and the adoption is easier for teams that are new.

And as usual, there is no strict black or white when it comes to consumer driven vs bi-directional. If you value the guaranteed behavior, start on a green field and already have closely collaborating teams, consumer driven with Pact might still be the thing for you. If you value the independence and lower learning curve more, consider a Pactflow subscription and go for bi-directional contract testing, but keep in mind that this needs a stronger focus on the API specification and maintenance of the spec files.

Hopefully this article gave you a better idea on how to use bi-directional contract testing, if you have trouble with the sample code don’t hesitate to reach out on the GitHub projects (discussion or issues). And if you have a solution for defining several responses based on status code or some other option, definitely reach out please!