# Using OpenTelemetry Tracing

This guide explains how your Quarkus application can utilize OpenTelemetry (OTel) to provide distributed tracing for interactive web applications.

This document is part of the Observability in Quarkus reference guide which features this and other observability related components.

| The OpenTelemetry Guide is available with signal independent information about the OpenTelemetry extension. If you search more information about OpenTelemetry Metrics, please refer to the OpenTelemetry Metrics Guide.

## Prerequisites

To complete this guide, you need:

- Roughly 15 minutes

- An IDE

- JDK 17+ installed with `JAVA_HOME` configured appropriately

- Apache Maven 3.9.9

- Docker and Docker Compose or Podman, and Docker Compose

- Optionally the Quarkus CLI if you want to use it

- Optionally Mandrel or GraalVM installed and configured appropriately if you want to build a native executable (or Docker if you use a native container build)

## Architecture

In this guide, we create a straightforward REST application to demonstrate distributed tracing.

## Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can skip right to the completed example.

Clone the Git repository: `git clone https://github.com/quarkusio/quarkus-quickstarts.git`, or download an archive.

The solution is located in the `opentelemetry-quickstart` directory.

## Creating the Maven project

First, we need a new project. Create a new project with the following command:

CLI

    quarkus create app org.acme:opentelemetry-quickstart \
        --extension='rest,quarkus-opentelemetry' \
        --no-code
    cd opentelemetry-quickstart

To create a Gradle project, add the `--gradle` or `--gradle-kotlin-dsl` option.

For more information about how to install and use the Quarkus CLI, see the Quarkus CLI guide.

Maven

    mvn io.quarkus.platform:quarkus-maven-plugin:3.20.2.1:create \
        -DprojectGroupId=org.acme \
        -DprojectArtifactId=opentelemetry-quickstart \
        -Dextensions='rest,quarkus-opentelemetry' \
        -DnoCode
    cd opentelemetry-quickstart

To create a Gradle project, add the `-DbuildTool=gradle` or `-DbuildTool=gradle-kotlin-dsl` option.

For Windows users:

- If using cmd, (don’t use backward slash `\` and put everything on the same line)

- If using Powershell, wrap `-D` parameters in double quotes e.g. `"-DprojectArtifactId=opentelemetry-quickstart"`

This command generates the Maven project and imports the `quarkus-opentelemetry` extension, which includes the default OpenTelemetry support, and a gRPC span exporter for OTLP.

If you already have your Quarkus project configured, you can add the `quarkus-opentelemetry` extension to your project by running the following command in your project base directory:

CLI

    quarkus extension add opentelemetry

Maven

    ./mvnw quarkus:add-extension -Dextensions='opentelemetry'

Gradle

    ./gradlew addExtension --extensions='opentelemetry'

This will add the following to your build file:

pom.xml

    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-opentelemetry</artifactId>
    </dependency>

build.gradle

    implementation("io.quarkus:quarkus-opentelemetry")

### Examine the Jakarta REST resource

Create a `src/main/java/org/acme/opentelemetry/TracedResource.java` file with the following content:

    package org.acme.opentelemetry;

    import jakarta.ws.rs.GET;
    import jakarta.ws.rs.Path;
    import jakarta.ws.rs.Produces;
    import jakarta.ws.rs.core.MediaType;
    import org.jboss.logging.Logger;

    @Path("/hello")
    public class TracedResource {

        private static final Logger LOG = Logger.getLogger(TracedResource.class);

        @GET
        @Produces(MediaType.TEXT_PLAIN)
        public String hello() {
            LOG.info("hello");
            return "hello";
        }
    }

Notice that there is no tracing specific code included in the application. By default, requests sent to this endpoint will be traced without any required code changes.

### Create the configuration

By default, the exporters will send out data in batches, using the gRPC protocol and endpoint `http://localhost:4317`.

If you need to change any of the default property values, here is an example on how to configure the default OTLP gRPC Exporter within the application, using the `src/main/resources/application.properties` file:

    quarkus.application.name=myservice (1)
    quarkus.otel.exporter.otlp.endpoint=http://localhost:4317 (2)
    quarkus.otel.exporter.otlp.headers=authorization=Bearer my_secret (3)
    quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, parentId=%X{parentId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n  (4)

    # Alternative to the console log
    quarkus.http.access-log.pattern="...traceId=%{X,traceId} spanId=%{X,spanId}" (5)

1 | All telemetry created from the application will include an OpenTelemetry `Resource` attribute indicating the telemetry was created by the `myservice` application. If not set, it will default to the artifact id.
2 | gRPC endpoint to send the telemetry. If not set, it will default to `http://localhost:4317`.
3 | Optional gRPC headers commonly used for authentication
4 | Add tracing information into log messages.
5 | You can also only put the trace info into the access log. In this case you must omit the info in the console log format.

We provide signal agnostic configurations for the connection related properties, meaning that you can use the same properties for both tracing and metrics when you set:

    quarkus.otel.exporter.otlp.endpoint=http://localhost:4317

If you need different configurations for each signal, you can use the specific properties:

    quarkus.otel.exporter.otlp.traces.endpoint=http://trace-uri:4317 (1)
    quarkus.otel.exporter.otlp.metrics.endpoint=http://metrics-uri:4317 (2)
    quarkus.otel.exporter.otlp.logs.endpoint=http://logs-uri:4317 (3)

1 | The endpoint for the traces exporter.
2 | The endpoint for the metrics exporter.
3 | The endpoint for the logs exporter.

If you need that your spans and logs to be exported directly as they finish (e.g. in a serverless environment / application), you can set this property to `true`. This replaces the default batching of data.

    quarkus.otel.simple=true

## Run the application

First we need to start a system to visualise the OpenTelemetry data. We have 2 options:

- Start an all-in-one Grafana OTel LGTM system for traces and metrics.

- Jaeger system just for traces.

### Grafana OTel LGTM option

- Take a look at: Getting Started with Grafana-OTel-LGTM.

This features a Quarkus Dev service including a Grafana for visualizing data, Loki to store logs, Tempo to store traces and Prometheus to store metrics. Also provides an OTel collector to receive the data.

### Jaeger to see traces option

Configure and start the OpenTelemetry Collector to receive, process and export telemetry data to Jaeger that will display the captured traces.

| Jaeger-all-in-one includes the Jaeger agent, an OTel collector, and the query service/UI. You do not need to install a separated collector. You can directly send the trace data to Jaeger (after enabling OTLP receivers there, see e.g. this blog entry for details).

Start the OpenTelemetry Collector and Jaeger system via the following `docker-compose.yml` file that you can launch via `docker-compose up -d`:

    version: "2"
    services:

      # Jaeger
      jaeger-all-in-one:
        image: jaegertracing/all-in-one:latest
        ports:
          - "16686:16686" # Jaeger UI
          - "14268:14268" # Receive legacy OpenTracing traces, optional
          - "4317:4317"   # OTLP gRPC receiver
          - "4318:4318"   # OTLP HTTP receiver
          - "14250:14250" # Receive from external otel-collector, optional
        environment:
          - COLLECTOR_OTLP_ENABLED=true

You should remove the optional ports you don’t need them.

### Start the application

Now we are ready to run our application. If using `application.properties` to configure the tracer:

CLI

    quarkus dev

Maven

    ./mvnw quarkus:dev

Gradle

    ./gradlew --console=plain quarkusDev

or if configuring the OTLP gRPC endpoint via JVM arguments:

CLI

    quarkus dev -Djvm.args="-Dquarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4317"

Maven

    ./mvnw quarkus:dev -Djvm.args="-Dquarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4317"

Gradle

    ./gradlew --console=plain quarkusDev -Djvm.args="-Dquarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4317"

With the OpenTelemetry Collector, the Jaeger system and the application running, you can make a request to the provided endpoint:

    $ curl http://localhost:8080/hello
    hello

When the first request has been submitted, you will be able to see the tracing information in the logs:

    10:49:02 INFO  traceId=, parentId=, spanId=, sampled= [io.quarkus] (main) Installed features: [cdi, opentelemetry, resteasy-client, resteasy, smallrye-context-propagation, vertx]
    10:49:03 INFO  traceId=17ceb8429b9f25b0b879fa1503259456, parentId=3125c8bee75b7ad6, spanId=58ce77c86dd23457, sampled=true [or.ac.op.TracedResource] (executor-thread-1) hello
    10:49:03 INFO  traceId=ad23acd6d9a4ed3d1de07866a52fa2df, parentId=, spanId=df13f5b45cf4d1e2, sampled=true [or.ac.op.TracedResource] (executor-thread-0) hello

Then visit the Jaeger UI to see the tracing information.

Hit `CTRL+C` or type `q` to stop the application.

### JDBC

The JDBC instrumentation will add a span for each JDBC queries done by your application, to enable it, add the following dependency to your build file:

pom.xml

    <dependency>
        <groupId>io.opentelemetry.instrumentation</groupId>
        <artifactId>opentelemetry-jdbc</artifactId>
    </dependency>

build.gradle

    implementation("io.opentelemetry.instrumentation:opentelemetry-jdbc")

As it uses a dedicated JDBC datasource wrapper, you must enable telemetry for your datasource:

    # enable tracing
    quarkus.datasource.jdbc.telemetry=true

    # configure datasource
    quarkus.datasource.db-kind=postgresql
    quarkus.datasource.jdbc.url=jdbc:postgresql://localhost:5432/mydatabase

## Additional configuration

Some use cases will require custom configuration of OpenTelemetry. These sections will outline what is necessary to properly configure it.

### ID Generator

The OpenTelemetry extension will use by default a random ID Generator when creating the trace and span identifier.

Some vendor-specific protocols need a custom ID Generator, you can override the default one by creating a producer. The OpenTelemetry extension will detect the `IdGenerator` CDI bean and will use it when configuring the tracer producer.

    @Singleton
    public class CustomConfiguration {

        /** Creates a custom IdGenerator for OpenTelemetry */
        @Produces
        @Singleton
        public IdGenerator idGenerator() {
            return AwsXrayIdGenerator.getInstance();
        }
    }

### Propagators

OpenTelemetry propagates cross-cutting concerns through propagators that will share an underlying `Context` for storing state and accessing data across the lifespan of a distributed transaction.

By default, the OpenTelemetry extension enables the W3C Trace Context and the W3C Baggage propagators, you can however choose any of the supported OpenTelemetry propagators by setting the `propagators` config that is described in the OpenTelemetry Configuration Reference.

#### Additional Propagators

- The `b3`, `b3multi`, `jaeger` and `ottrace` propagators will need the trace-propagators extension to be added as a dependency to your project.

pom.xml

    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-extension-trace-propagators</artifactId>
    </dependency>

build.gradle

    implementation("io.opentelemetry:opentelemetry-extension-trace-propagators")

- The `xray` propagator will need the aws extension to be added as a dependency to your project.

pom.xml

    <dependency>
        <groupId>io.opentelemetry.contrib</groupId>
        <artifactId>opentelemetry-aws-xray-propagator</artifactId>
    </dependency>

build.gradle

    implementation("io.opentelemetry.contrib:opentelemetry-aws-xray-propagator")

#### Customise Propagator

To customise the propagation header you can implement the `TextMapPropagatorCustomizer` interface. This can be used, as an example, to restrict propagation of OpenTelemetry trace headers and prevent potentially sensitive data to be sent to third party systems.

    /**
     * /**
     * Meant to be implemented by a CDI bean that provides arbitrary customization for the TextMapPropagator
     * that are to be registered with OpenTelemetry
     */
    public interface TextMapPropagatorCustomizer {

        TextMapPropagator customize(Context context);

        interface Context {
            TextMapPropagator propagator();

            ConfigProperties otelConfigProperties();
        }
    }

### Resource

See the main OpenTelemetry Guide resources section.

#### End User attributes

When enabled, Quarkus adds OpenTelemetry End User attributes as Span attributes. Before you enable this feature, verify that Quarkus Security extension is present and configured. More information about the Quarkus Security can be found in the Quarkus Security overview.

The attributes are only added when authentication has already happened on a best-efforts basis. Whether the End User attributes are added as Span attributes depends on authentication and authorization configuration of your Quarkus application. If you create custom Spans prior to the authentication, Quarkus cannot add the End User attributes to them. Quarkus is only able to add the attributes to the Span that is current after the authentication has been finished. Another important consideration regarding custom Spans is active CDI request context that is used to propagate Quarkus `SecurityIdentity`. In principle, Quarkus is able to add the End User attributes when the CDI request context has been activated for you before the custom Spans are created.

    quarkus.otel.traces.eusp.enabled=true (1)
    quarkus.http.auth.proactive=true (2)

1 | Enable the End User Attributes feature so that the `SecurityIdentity` principal and roles are added as Span attributes. The End User attributes are personally identifiable information, therefore make sure you want to export them before you enable this feature.
2 | Optionally enable proactive authentication. The best possible results are achieved when proactive authentication is enabled because the authentication happens sooner. A good way to determine whether proactive authentication should be enabled in your Quarkus application is to read the Quarkus Proactive authentication guide.

| This feature is not supported when a custom Jakarta REST SecurityContexts is used.

### Sampler

A sampler decides whether a trace should be discarded or forwarded, effectively managing noise and reducing overhead by limiting the number of collected traces sent to the collector.

Quarkus comes equipped with a built-in sampler, and you also have the option to create your custom sampler.

To use the built-in sampler, you can configure it by setting the desired sampler parameters as detailed in the OpenTelemetry Configuration Reference. As an example, you can configure the sampler to retain 50% of the traces:

    # build time property only:
    quarkus.otel.traces.sampler=traceidratio
    # Runtime property:
    quarkus.otel.traces.sampler.arg=0.5

| An interesting use case for the sampler is to activate and deactivate tracing export at runtime, according to this example:
    # build time property only:
    quarkus.otel.traces.sampler=traceidratio
    # On (default). All traces are exported:
    quarkus.otel.traces.sampler.arg=1.0
    # Off. No traces are exported:
    quarkus.otel.traces.sampler.arg=0.0

If you need to use a custom sampler there are now 2 different ways:

#### Sampler CDI Producer

You can create a sampler CDI producer. The Quarkus OpenTelemetry extension will detect the `Sampler` CDI bean and will use it when configuring the Tracer.

    @Singleton
    public class CustomConfiguration {

        /** Creates a custom sampler for OpenTelemetry */
        @Produces
        @Singleton
        public Sampler sampler() {
            return JaegerRemoteSampler.builder()
            .setServiceName("my-service")
            .build();
        }
    }

#### OTel Sampler SPI

This will use the SPI hooks available with the OTel Autoconfiguration. You can create a simple Sampler class:

    public class CustomSPISampler implements Sampler {
        @Override
        public SamplingResult shouldSample(Context context,
                String s,
                String s1,
                SpanKind spanKind,
                Attributes attributes,
                List<LinkData> list) {
            // Do some sampling here
            return Sampler.alwaysOn().shouldSample(context, s, s1, spanKind, attributes, list);
        }

        @Override
        public String getDescription() {
            return "custom-spi-sampler-description";
        }
    }

Then a Sampler Provider:

    public class CustomSPISamplerProvider implements ConfigurableSamplerProvider {
        @Override
        public Sampler createSampler(ConfigProperties configProperties) {
            return new CustomSPISampler();
        }

        @Override
        public String getName() {
            return "custom-spi-sampler";
        }
    }

Write the SPI loader text file at `resources/META-INF/services` with name `io.opentelemetry.sdk.autoconfigure.spi.traces.ConfigurableSamplerProvider` containing the full qualified name of the `CustomSPISamplerProvider` class.

Then activate on the configuration:

    quarkus.otel.traces.sampler=custom-spi-sampler

As you can see, CDI is much simpler to work with.

## Additional instrumentation

Some Quarkus extensions will require additional code to ensure traces are propagated to subsequent execution. These sections will outline what is necessary to propagate traces across process boundaries.

The instrumentation documented in this section has been tested with Quarkus and works in both standard and native mode.

### CDI

Annotating a method in any CDI aware bean with the `io.opentelemetry.instrumentation.annotations.WithSpan` annotation will create a new Span and establish any required relationships with the current Trace context.

Annotating a method in any CDI aware bean with the `io.opentelemetry.instrumentation.annotations.AddingSpanAttributes` will not create a new span but will add annotated method parameters to attributes in the current span.

If a method is annotated by mistake with `@AddingSpanAttributes` and `@WithSpan` annotations, the `@WithSpan` annotation will take precedence.

Method parameters can be annotated with the `io.opentelemetry.instrumentation.annotations.SpanAttribute` annotation to indicate which method parameters should be part of the span. The parameter name can be customized as well.

Example:

    @ApplicationScoped
    class SpanBean {
        @WithSpan
        void span() {

        }

        @WithSpan("name")
        void spanName() {

        }

        @WithSpan(kind = SERVER)
        void spanKind() {

        }

        @WithSpan
        void spanArgs(@SpanAttribute(value = "arg") String arg) {

        }

        @AddingSpanAttributes
        void addArgumentToExistingSpan(@SpanAttribute(value = "arg") String arg) {

        }
    }

### Available OpenTelemetry CDI injections

As per MicroProfile Telemetry Tracing specification, Quarkus supports the CDI injections of the following classes:

- `io.opentelemetry.api.OpenTelemetry`

- `io.opentelemetry.api.trace.Tracer`

- `io.opentelemetry.api.trace.Span`

- `io.opentelemetry.api.baggage.Baggage`

You can inject these classes in any CDI enabled bean. For instance, the `Tracer` is particularly useful to start custom spans:

    @Inject
    Tracer tracer;

    ...

    public void tracedWork() {
        Span span = tracer.spanBuilder("My custom span")
            .setAttribute("attr", "attr.value")
            .setParent(Context.current().with(Span.current()))
            .setSpanKind(SpanKind.INTERNAL)
            .startSpan();

        // traced work

        span.end();
    }

### Mutiny

Methods returning reactive types can also be annotated with `@WithSpan` and `@AddingSpanAttributes` to create a new span or add attributes to the current span.

If you need to create spans manually within a mutiny pipeline, use `wrapWithSpan` method from `io.quarkus.opentelemetry.runtime.tracing.mutiny.MutinyTracingHelper`.

Example. Assuming you have the following pipeline:

    Uni<String> uni = Uni.createFrom().item("hello")
            //start trace here
            .onItem().transform(item -> item + " world")
            .onItem().transform(item -> item + "!")
            //end trace here
            .subscribe().with(
                    item -> System.out.println("Received: " + item),
                    failure -> System.out.println("Failed with " + failure)
            );

wrap it like this:

    import static io.quarkus.opentelemetry.runtime.tracing.mutiny.MutinyTracingHelper.wrapWithSpan;
    ...
    @Inject
    Tracer tracer;
    ...
    Context context = Context.current();
    Uni<String> uni = Uni.createFrom().item("hello")
            .transformToUni(m -> wrapWithSpan(tracer, Optional.of(context), "my-span-name",
                                    Uni.createFrom().item(m)
                                        .onItem().transform(item -> item + " world")
                                        .onItem().transform(item -> item + "!")
            ))
            .subscribe().with(
                    item -> System.out.println("Received: " + item),
                    failure -> System.out.println("Failed with " + failure)
            );

for multi-pipelines it works similarly:

    Multi.createFrom().items("Alice", "Bob", "Charlie")
            .transformToMultiAndConcatenate(m -> TracingHelper.withTrace("my-span-name",
                                    Multi.createFrom().item(m)
                                        .onItem().transform(name -> "Hello " + name)
            ))
            .subscribe().with(
                    item -> System.out.println("Received: " + item),
                    failure -> System.out.println("Failed with " + failure)
            );

Instead of `transformToMultiAndConcatenate` you can use `transformToMultiAndMerge` if you don’t care about the order of the items.

### Quarkus Messaging - Kafka

When using the Quarkus Messaging extension for Kafka, we are able to propagate the span into the Kafka Record with:

    TracingMetadata tm = TracingMetadata.withPrevious(Context.current());
    Message out = Message.of(...).withMetadata(tm);

The above creates a `TracingMetadata` object we can add to the `Message` being produced, which retrieves the OpenTelemetry `Context` to extract the current span for propagation.

### Quarkus Security Events

Quarkus supports exporting of the Security events as OpenTelemetry Span events.

    quarkus.otel.security-events.enabled=true (1)

1 | Export Quarkus Security events as OpenTelemetry Span events.

## Exporters

See the main OpenTelemetry Guide exporters section.

## Quarkus core extensions instrumented with OpenTelemetry tracing

- `quarkus-agroal`

- `quarkus-grpc`

- `quarkus-redis-client`

- `quarkus-rest`

- `quarkus-rest-client`

- `quarkus-rest-client-jaxrs`

- `quarkus-resteasy`

- `quarkus-resteasy-client`

- `quarkus-scheduler`

- `quarkus-smallrye-graphql`

- `quarkus-mongodb-client`

- `quarkus-messaging`

  - AMQP 1.0

  - RabbitMQ

  - Kafka

  - Pulsar

- `quarkus-vertx` (http requests)

- `websockets-next`

### Disable parts of the automatic tracing

Automatic tracing instrumentation parts can be disabled by setting `quarkus.otel.instrument.*` properties to `false`.

Examples:

    quarkus.otel.instrument.grpc=false
    quarkus.otel.instrument.messaging=false
    quarkus.otel.instrument.resteasy-client=false
    quarkus.otel.instrument.rest=false
    quarkus.otel.instrument.resteasy=false

### Disabling Specific Traces for Application Endpoints

You can use the `quarkus.otel.traces.suppress-application-uris` property to exclude specific endpoints from being traced.

#### Example Configuration

    # application.properties
    quarkus.otel.traces.suppress-application-uris=trace,ping,people*

This configuration will:

- Disable tracing for the `/trace` URI.

- Disable tracing for the `/ping` URI.

- Disable tracing for the `/people` URI and all subpaths, such as `/people/1` and `/people/1/cars`.

| If you are using `quarkus.http.root-path`, ensure you include the root path in the configuration.

## OpenTelemetry Configuration Reference

See the main OpenTelemetry Guide configuration reference.