Microprofile Fault Tolerance Retry in Action
Tuesday, March 02, 2021 |As part of some of my recent work, I’ve gotten some exposure to some Microprofile specs I’ve not had the opportunity or need to use. One of those is Fault Tolerance. I was curious to see it action, so I’ve cobbled together this simple example that demonstrates some of that spec’s features, namely retries and fallback.
It should be noted that the Fault Tolerance spec actually provides several mechanisms to improve the reliability of a distributed system including timeouts, retries, bulkheads, circuit breakers, and fallbacks. We are only going to look at retries and fallbacks here. Perhaps in a future post we’ll explore the others.
To start, let’s describe our (contrived) scenario and show the code. We’re going to implement a simple REST service that calls an external service. In this case, we’re going to submit a search request to Google. (We won’t be using any API for this, just a simple GET request. There’s no need for anything more complicated at this point). To that end, here’s our gem:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
@Path("/failing")
public class FailingResource {
@Inject
@RestClient
GoogleClient client;
@GET
@Produces(MediaType.TEXT_PLAIN)
@Retry(maxRetries = 2,
delay = 200,
jitter = 100)
@Fallback(fallbackMethod = "doWorkFallback")
public String hello() throws URISyntaxException {
return client.search("microprofile");
}
private String doWorkFallback() {
return "fallback";
}
}
It’s a pretty vanilla resource. On the resource method itself, note that we have two additional annotionations: @Retry and @Fallback. Using @Retry, we specifiy how many times the method should be called (maxRetries), how long to wait between invocations (delay), as well as a random "jitter", or variation, in the delay (jitter). Should maxRetries be exceeded, @Fallback comes into play, instructing the system to call doWorkFallback().
In this example, we’re also using the Microprofile REST Client. Note the injection of a GoogleClient:
1
2
3
4
5
6
7
@RegisterRestClient
@Path("/")
public interface GoogleClient {
@GET
@Path("search")
String search(@QueryParam("q") String text);
}
I don’t want to spend too much time here, so I’ll just note that the MP REST Client allows us to create a type-safe REST client as an interface, with which "the system" will then create a concrete instance that does the actual work of making the remote calls. We configure that client using yet another specification, Microprofile Config:
1
com.steeplesoft.GoogleClient/mp-rest/url=https://www.google.com
The client will make requests against https://www.google.com, but, for testing, we’ll need to change that. We’ll see how to do that in a bit.
As in my last post, we’re going to test this using Wiremock. Before we jump into the code, though, let’s talk about how we need to go about testing this. Obviously, the point of the exercise to see Fault Tolerance’s retry and fallback in action. From what we saw in the last post, though, it seems we configure Wiremock to return certain response for each request to a given endpoint. While that is certainly true, Wiremock does allow us to change what the response is based on something they call scenarios.
A scenarios is "essentially a state machine whose states can be arbitrarily assigned." They are arbitrarily named using string names, but always start with Scenario.STARTED. Using the Wiremock API, we can advance the scenario from one state to another based on various aspects of the request (such as reqest body, query params, etc). In our example, though, we just want it to fail a certain number of times, then, possibly, not fail. Let’s see what that looks like.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
@QuarkusTest
@QuarkusTestResource(MockServer.class)
public class FailingResourceTest {
public static final String SCENARIO_NAME = "Failing Resource";
public static final String[] STATES = new String[]{"one", "two", "three", "four", "five"};
public static final String REQUEST_URL = "/search?q=microprofile";
@Test
public void testSuccessfulRetry() throws URISyntaxException {
reset();
stubFor(get(urlEqualTo(REQUEST_URL))
.inScenario(SCENARIO_NAME)
.whenScenarioStateIs(Scenario.STARTED)
.willReturn(aResponse().withStatus(500))
.willSetStateTo(STATES[0]));
stubFor(get(urlEqualTo(REQUEST_URL))
.inScenario(SCENARIO_NAME)
.whenScenarioStateIs(STATES[0])
.willReturn(aResponse().withStatus(500))
.willSetStateTo(STATES[1]));
stubFor(get(urlEqualTo(REQUEST_URL))
.inScenario(SCENARIO_NAME)
.whenScenarioStateIs(STATES[1])
.willReturn(aResponse().withBody("success")));
given()
.when()
.get("/failing")
.then()
.statusCode(200)
.body(is("success"));
}
}
While it’s overkill, we call reset() at the start to…wait for it… reset the Wiremock state, making sure the stubs and scenarios are in a clean, known state. The need for that will become apparent shortly. The stubFor() calls follow this logic:
-
For requests to
/search?q=microprofile, if the scenario isSTARTED, return a500error, and set the state toone. -
For requests to
/search?q=microprofile, if the scenario isone, return a500error, and set the state totwo. -
For requests to
/search?q=microprofile, if the scenario istwo, return a200, with a response body ofsuccess.
Now, using RestAssured, we call our resource, which calls the mocked server using the MP Rest Client we configured above. Before we can do that, though, we need to setup and start Wiremock:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public class MockServer implements QuarkusTestResourceLifecycleManager {
private WireMockServer wireMockServer;
@Override
public Map<String, String> start() {
wireMockServer = new WireMockServer();
wireMockServer.start();
return Map.of("com.steeplesoft.GoogleClient/mp-rest/url", wireMockServer.baseUrl());
}
@Override
public void stop() {
wireMockServer.stop();
}
}
Since we’re stubbing the responses in our test, our setup is pretty simple. The super important part here, though, is the Map we return. Remember how we configure the REST Client using Microprofile Config? We can override the value of that configuration property by adding a key of the same name to the map, but providing the URL representing our mock server: wireMockServer.baseUrl(). Now if you run your test, you should get a green test. Huzzah!
But what about the retry? The fallback? Let’s add another test, with a slightly different scenario setup. Here’s where reset() is important ;)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Test
public void testUnsuccessfulRetry() {
reset();
stubFor(get(urlEqualTo(REQUEST_URL))
.inScenario(SCENARIO_NAME)
.whenScenarioStateIs(Scenario.STARTED)
.willReturn(aResponse().withStatus(500)));
given()
.when()
.get("/failing")
.then()
.statusCode(200)
.body(is("fallback"));
}
In this scenario, we will always return a 500 error since we don’t really care if "the server" ever recovers. We just want to attempts to fail over to our fallback method, in which case our resource returns "fallback".
There is much, much more to the Fault Tolerance spec, but that gives you a taste of what retry and fallback looks like. You can find the complete project here.
My name is Jason Lee. I am a software developer living in the middle of Oklahoma. I’ve been a professional developer since 1997,
using a variety of languages, including Java, Javascript, PHP, Python, Delphi, and even a bit of C#. I currently work for Red Hat
on the WildFly/EAP team, where, among other things, I maintain integrations for some MicroProfile specs, OpenTelemetry, Micrometer,
Jakarta Faces, and Bean Validation. (Full resume 