Inter-container Communications with Testcontainers

Monday, February 19, 2024 |

I recently found myself in need of having two different Testcontainers communicate with each other. To someone more familiar with Docker, the solution might have been more obvious, but, alas, I am not that man. :P After asking in the Testcontainer Slack, I got a pointer, so I thought I’d share it here in case it might help someone else.

To be specific, I needed to have the OpenTelemetry Collector pushing trace data to Jaeger so that I could more easily test some WildFly changes. (There might be a better way, but this is working for now, and incremental improvement is the name of the game ;). The trick is to create a Network that the two containers will share. Fortunately, Testcontainers has a network defined for use already: Network.SHARED. In my case, I don’t need anything fancy, so I can just use this. If you have more complicated needs, the Javadoc should help you with that.

With the Network defined, I just need to configure each container. In this scenario, I need to set up the Jaeger container with both the Network as well as a network alias, or host name, by which the Otel Collector can address it. For example:

1
2
3
new JaegerContainer()
    .withNetwork(Network.SHARED) // <---- This
    .withNetworkAliases("jaeger");

(See below for the full class)

Now, for the collector:

1
2
3
4
5
6
new OpenTelemetryCollectorContainer()
    .withNetwork(Network.SHARED) // <---- This
    .withCopyToContainer(MountableFile.forClasspathResource(
        "org/wildfly/test/integration/observability/container/otel-collector-config.yaml"),
            OpenTelemetryCollectorContainer.OTEL_COLLECTOR_CONFIG_YAML)
    .withCommand("--config " + OpenTelemetryCollectorContainer.OTEL_COLLECTOR_CONFIG_YAML);

(Full class also below)

The config file is a classpath resource, the looks something like this:

1
2
3
4
5
6
7
...
exporters:
  otlp:
    endpoint: http://jaeger:4317
    tls:
      insecure: true
...

Notice in the otlp exporter, I simply refer to the other container by the configured hostname, and the Docker network figures everthing else out. Note also that I don’t have to use the mapped port from the Testcontainer, as connection will use the exposed port configured (4317) inside the Docker network, so there’s no chance of conflicts with the host machine.

And, fundamentally, that’s all there is to it. I can now have my WildFly instance push traces to the OpenTelemetryCollectorContainer, which forwards those traces, via OTLP, to the JaegerContainer, and I can view those traces via the Jaeger UI (or the undocumented REST API, which is what I’m actually using in my tests. Sssh… Don’t tell anyone :).

Hope that helps!

JaegerContainer.java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
import java.util.List;
import jakarta.ws.rs.client.Client;
import jakarta.ws.rs.client.ClientBuilder;
import jakarta.ws.rs.client.WebTarget;

import org.junit.Assert;
import org.testcontainers.containers.Network;
import org.testcontainers.containers.wait.strategy.Wait;
import org.wildfly.common.annotation.NotNull;
import org.wildfly.test.integration.observability.opentelemetry.jaeger.JaegerResponse;
import org.wildfly.test.integration.observability.opentelemetry.jaeger.JaegerTrace;

/*
 * This class is really intended to be called ONLY from OpenTelemetryCollectorContainer. Any test working with
 * tracing data should be passing through the otel collector and any methods on its Container.
 */
class JaegerContainer extends BaseContainer<JaegerContainer> {
    private static JaegerContainer INSTANCE = null;

    public static final int PORT_JAEGER_QUERY = 16686;
    public static final int PORT_JAEGER_OTLP = 4317;

    private String jaegerEndpoint;

    private JaegerContainer() {
        super("Jaeger", "jaegertracing/all-in-one", "latest",
                List.of(PORT_JAEGER_QUERY, PORT_JAEGER_OTLP),
                List.of(Wait.forHttp("/").forPort(PORT_JAEGER_QUERY)));
    }

    @NotNull
    public static synchronized JaegerContainer getInstance() {
        if (INSTANCE == null) {
            INSTANCE = new JaegerContainer()
                    .withNetwork(Network.SHARED)
                    .withNetworkAliases("jaeger")
                    .withEnv("JAEGER_DISABLED", "true");
            INSTANCE.start();
        }

        return INSTANCE;
    }

    @Override
    public void start() {
        super.start();
        jaegerEndpoint = "http://localhost:" + getMappedPort(PORT_JAEGER_QUERY);
    }

    @Override
    public synchronized void stop() {
        INSTANCE = null;
        super.stop();
    }

    List<JaegerTrace> getTraces(String serviceName) throws InterruptedException {
        try (Client client = ClientBuilder.newClient()) {
            waitForDataToAppear(serviceName);
            String jaegerUrl = jaegerEndpoint + "/api/traces?service=" + serviceName;
            JaegerResponse jaegerResponse = client.target(jaegerUrl).request().get().readEntity(JaegerResponse.class);
            return jaegerResponse.getData();
        }
    }

    private void waitForDataToAppear(String serviceName) {
        try (Client client = ClientBuilder.newClient()) {
            String uri = jaegerEndpoint + "/api/services";
            WebTarget target = client.target(uri);
            boolean found = false;
            int count = 0;
            while (count < 10) {
                String response = target.request().get().readEntity(String.class);
                if (response.contains(serviceName)) {
                    found = true;
                    break;
                }
                count++;
                try {
                    Thread.sleep(500);
                } catch (InterruptedException e) {
                    //
                }
            }

            Assert.assertTrue("Expected service name not found", found);
        }
    }
}

OpenTelemetryCollectorContainer.java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
import java.util.Collections;
import java.util.List;

import org.testcontainers.containers.Network;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.utility.MountableFile;
import org.wildfly.common.annotation.NotNull;
import org.wildfly.test.integration.observability.opentelemetry.jaeger.JaegerTrace;

public class OpenTelemetryCollectorContainer extends BaseContainer<OpenTelemetryCollectorContainer> {
    private static OpenTelemetryCollectorContainer INSTANCE = null;
    private static JaegerContainer jaegerContainer;

    public static final int OTLP_GRPC_PORT = 4317;
    public static final int OTLP_HTTP_PORT = 4318;
    public static final int PROMETHEUS_PORT = 49152;
    public static final int HEALTH_CHECK_PORT = 13133;

    public static final String OTEL_COLLECTOR_CONFIG_YAML = "/etc/otel-collector-config.yaml";

    private String otlpGrpcEndpoint;
    private String otlpHttpEndpoint;
    private String prometheusUrl;


    private OpenTelemetryCollectorContainer() {
        super("OpenTelemetryCollector",
                "otel/opentelemetry-collector",
                "0.93.0",
                List.of(OTLP_GRPC_PORT, OTLP_HTTP_PORT, HEALTH_CHECK_PORT, PROMETHEUS_PORT),
                List.of(Wait.forHttp("/").forPort(HEALTH_CHECK_PORT)));
    }

    @NotNull
    public static synchronized OpenTelemetryCollectorContainer getInstance() {
        if (INSTANCE == null) {
            jaegerContainer = JaegerContainer.getInstance();

            INSTANCE = new OpenTelemetryCollectorContainer()
                    .withNetwork(Network.SHARED)
                    .withCopyToContainer(MountableFile.forClasspathResource(
                                    "org/wildfly/test/integration/observability/container/otel-collector-config.yaml"),
                            OpenTelemetryCollectorContainer.OTEL_COLLECTOR_CONFIG_YAML)
                    .withCommand("--config " + OpenTelemetryCollectorContainer.OTEL_COLLECTOR_CONFIG_YAML);
            INSTANCE.start();
        }
        return INSTANCE;
    }

    @Override
    public void start() {
        super.start();
        otlpGrpcEndpoint = "http://localhost:" + getMappedPort(OTLP_GRPC_PORT);
        otlpHttpEndpoint = "http://localhost:" + getMappedPort(OTLP_HTTP_PORT);
        prometheusUrl = "http://localhost:" + getMappedPort(PROMETHEUS_PORT) + "/metrics";
    }

    @Override
    public synchronized void stop() {
        if (jaegerContainer != null) {
            jaegerContainer.stop();
        }
        INSTANCE = null;
        super.stop();
    }

    public String getOtlpGrpcEndpoint() {
        return otlpGrpcEndpoint;
    }

    public String getOtlpHttpEndpoint() {
        return otlpHttpEndpoint;
    }

    public String getPrometheusUrl() {
        return prometheusUrl;
    }

    public List<JaegerTrace> getTraces(String serviceName) throws InterruptedException {
        return (jaegerContainer != null ? jaegerContainer.getTraces(serviceName) : Collections.emptyList());
    }
}

Inter-container Communications with Testcontainers

JaegerContainer.java

OpenTelemetryCollectorContainer.java

Search

Quotes

About

My Links

Publications