|
| 1 | +[[apm-open-telemetry-direct]] |
| 2 | += Upstream OpenTelemetry Collectors and language SDKs |
| 3 | + |
| 4 | +[NOTE] |
| 5 | +==== |
| 6 | +This is one of <<apm-open-telemetry,several approaches>> you can use to integrate Elastic with OpenTelemetry. |
| 7 | +
|
| 8 | +*To compare approaches and choose the best approach for your use case, refer to <<apm-open-telemetry,Use OpenTelemetry with APM>>.* |
| 9 | +==== |
| 10 | + |
| 11 | +The {stack} natively supports the OpenTelemetry protocol (OTLP). |
| 12 | +This means trace data and metrics collected from your applications and infrastructure can |
| 13 | +be sent directly to the {stack}. |
| 14 | + |
| 15 | +* Send data to the {stack} from an upstream <<apm-connect-open-telemetry-collector,OpenTelemetry Collector>> |
| 16 | +* Send data to the {stack} from an upstream <<apm-instrument-apps-otel,OpenTelemetry language SDK>> |
| 17 | + |
| 18 | +[discrete] |
| 19 | +[[apm-connect-open-telemetry-collector]] |
| 20 | +== Send data from an upstream OpenTelemetry Collector |
| 21 | + |
| 22 | +Connect your OpenTelemetry Collector instances to Elastic {observability} using the OTLP exporter: |
| 23 | + |
| 24 | +[source,yaml] |
| 25 | +---- |
| 26 | +receivers: <1> |
| 27 | + # ... |
| 28 | + otlp: |
| 29 | + protocols: |
| 30 | + grpc: |
| 31 | + endpoint: 0.0.0.0:4317 |
| 32 | + http: |
| 33 | + endpoint: 0.0.0.0:4318 |
| 34 | +processors: <2> |
| 35 | + # ... |
| 36 | + memory_limiter: |
| 37 | + check_interval: 1s |
| 38 | + limit_mib: 2000 |
| 39 | + batch: |
| 40 | +
|
| 41 | +exporters: |
| 42 | + debug: |
| 43 | + verbosity: detailed <3> |
| 44 | + otlp: <4> |
| 45 | + # Elastic APM server https endpoint without the "https://" prefix |
| 46 | + endpoint: "${env:ELASTIC_APM_SERVER_ENDPOINT}" <5> <7> |
| 47 | + headers: |
| 48 | + # Elastic APM Server secret token |
| 49 | + Authorization: "Bearer ${env:ELASTIC_APM_SECRET_TOKEN}" <6> <7> |
| 50 | +
|
| 51 | +service: |
| 52 | + pipelines: |
| 53 | + traces: |
| 54 | + receivers: [otlp] |
| 55 | + processors: [..., memory_limiter, batch] |
| 56 | + exporters: [debug, otlp] |
| 57 | + metrics: |
| 58 | + receivers: [otlp] |
| 59 | + processors: [..., memory_limiter, batch] |
| 60 | + exporters: [debug, otlp] |
| 61 | + logs: <8> |
| 62 | + receivers: [otlp] |
| 63 | + processors: [..., memory_limiter, batch] |
| 64 | + exporters: [debug, otlp] |
| 65 | +---- |
| 66 | +<1> The receivers, like the |
| 67 | +https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver[OTLP receiver], that forward data emitted by APM agents, or the https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver[host metrics receiver]. |
| 68 | +<2> We recommend using the https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md[Batch processor] and the https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md[memory limiter processor]. For more information, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md#recommended-processors[recommended processors]. |
| 69 | +<3> The https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/debugexporter[debug exporter] is helpful for troubleshooting, and supports configurable verbosity levels: `basic` (default), `normal`, and `detailed`. |
| 70 | +<4> Elastic {observability} endpoint configuration. |
| 71 | +APM Server supports a ProtoBuf payload via both the OTLP protocol over gRPC transport https://opentelemetry.io/docs/specs/otlp/#otlpgrpc[(OTLP/gRPC)] |
| 72 | +and the OTLP protocol over HTTP transport https://opentelemetry.io/docs/specs/otlp/#otlphttp[(OTLP/HTTP)]. |
| 73 | +To learn more about these exporters, see the OpenTelemetry Collector documentation: |
| 74 | +https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter[OTLP/HTTP Exporter] or |
| 75 | +https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter[OTLP/gRPC exporter]. |
| 76 | +When adding an endpoint to an existing configuration an optional name component can be added, like `otlp/elastic`, to distinguish endpoints as described in the https://opentelemetry.io/docs/collector/configuration/#basics[OpenTelemetry Collector Configuration Basics]. |
| 77 | +<5> Hostname and port of the APM Server endpoint. For example, `elastic-apm-server:8200`. |
| 78 | +<6> Credential for Elastic APM <<apm-secret-token,secret token authorization>> (`Authorization: "Bearer a_secret_token"`) or <<apm-api-key,API key authorization>> (`Authorization: "ApiKey an_api_key"`). |
| 79 | +<7> Environment-specific configuration parameters can be conveniently passed in as environment variables documented https://opentelemetry.io/docs/collector/configuration/#environment-variables[here] (e.g. `ELASTIC_APM_SERVER_ENDPOINT` and `ELASTIC_APM_SECRET_TOKEN`). |
| 80 | +<8> preview:[] To send OpenTelemetry logs to {stack} version 8.0+, declare a `logs` pipeline. |
| 81 | + |
| 82 | +You're now ready to export traces and metrics from your services and applications. |
| 83 | + |
| 84 | +TIP: When using the OpenTelemetry Collector, you should always prefer sending data via the https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter[`OTLP` exporter] to an Elastic APM Server. |
| 85 | +Other methods, like using the https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter[`elasticsearch` exporter] to send data directly to {es} will send data to the {stack}, |
| 86 | +but will bypass all of the validation and data processing that the APM Server performs. |
| 87 | +In addition, your data will not be viewable in the {kib} {observability} apps if you use the `elasticsearch` exporter. |
| 88 | + |
| 89 | +[discrete] |
| 90 | +[[apm-instrument-apps-otel]] |
| 91 | +== Send data from an upstream OpenTelemetry SDK |
| 92 | + |
| 93 | +[NOTE] |
| 94 | +==== |
| 95 | +This document outlines how to send data directly from an upstream OpenTelemetry SDK to APM Server, which is appropriate when getting started. However, in many cases you should use the OpenTelemetry SDK to send data to an OpenTelemetry Collector that processes and exports data to APM Server. Read more about when and how to use a collector in the https://opentelemetry.io/docs/collector/#when-to-use-a-collector[OpenTelemetry documentation]. |
| 96 | +==== |
| 97 | + |
| 98 | +To export traces and metrics to APM Server, instrument your services and applications |
| 99 | +with the OpenTelemetry API, SDK, or both. For example, if you are a Java developer, you need to instrument your Java app with the |
| 100 | +https://github.com/open-telemetry/opentelemetry-java-instrumentation[OpenTelemetry agent for Java]. |
| 101 | +See the https://opentelemetry.io/docs/instrumentation/[OpenTelemetry Instrumentation guides] to download the |
| 102 | +OpenTelemetry agent or SDK for your language. |
| 103 | + |
| 104 | +Define environment variables to configure the OpenTelemetry agent or SDK and enable communication with Elastic APM. |
| 105 | +For example, if you are instrumenting a Java app, define the following environment variables: |
| 106 | + |
| 107 | +[source,bash] |
| 108 | +---- |
| 109 | +export OTEL_RESOURCE_ATTRIBUTES=service.name=checkoutService,service.version=1.1,deployment.environment=production |
| 110 | +export OTEL_EXPORTER_OTLP_ENDPOINT=<YOUR_APM_SERVER_URL>:8200 |
| 111 | +export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer an_apm_secret_token" |
| 112 | +export OTEL_METRICS_EXPORTER="otlp" \ |
| 113 | +export OTEL_LOGS_EXPORTER="otlp" \ <1> |
| 114 | +java -javaagent:/path/to/opentelemetry-javaagent-all.jar \ |
| 115 | + -classpath lib/*:classes/ \ |
| 116 | + com.mycompany.checkout.CheckoutServiceServer |
| 117 | +---- |
| 118 | +<1> preview:[] The OpenTelemetry logs intake via APM Server is currently in technical preview. |
| 119 | + |
| 120 | +|=== |
| 121 | + |
| 122 | +| `OTEL_RESOURCE_ATTRIBUTES` | Fields that describe the service and the environment that the service runs in. See |
| 123 | +<<apm-open-telemetry-resource-attributes,resource attributes>> for more information. |
| 124 | + |
| 125 | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | APM Server URL. The host and port that APM Server listens for events on. |
| 126 | + |
| 127 | +| `OTEL_EXPORTER_OTLP_HEADERS` a| Authorization header that includes the Elastic APM Secret token or API key: `"Authorization=Bearer an_apm_secret_token"` or `"Authorization=ApiKey an_api_key"`. |
| 128 | + |
| 129 | +For information on how to format an API key, see |
| 130 | +{observability-guide}/apm-api-key.html[API keys]. |
| 131 | + |
| 132 | +Please note the required space between `Bearer` and `an_apm_secret_token`, and `ApiKey` and `an_api_key`. |
| 133 | + |
| 134 | +NOTE: If you are using a version of the Python OpenTelemetry agent _before_ 1.27.0, the content of the header _must_ be URL-encoded. You can use the Python standard library's `urllib.parse.quote` function to encode the content of the header. |
| 135 | + |
| 136 | +| `OTEL_METRICS_EXPORTER` | Metrics exporter to use. See https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection[exporter selection] for more information. |
| 137 | + |
| 138 | +| `OTEL_LOGS_EXPORTER` | Logs exporter to use. See https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection[exporter selection] for more information. |
| 139 | + |
| 140 | +|=== |
| 141 | + |
| 142 | +You are now ready to collect traces and <<apm-open-telemetry-collect-metrics,metrics>> before <<apm-open-telemetry-verify-metrics,verifying metrics>> |
| 143 | +and <<apm-open-telemetry-visualize,visualizing metrics>> in {kib}. |
| 144 | + |
| 145 | +[discrete] |
| 146 | +[[apm-open-telemetry-proxy-apm]] |
| 147 | +== Proxy requests to APM Server |
| 148 | + |
| 149 | +APM Server supports both the https://opentelemetry.io/docs/specs/otlp/#otlpgrpc[OTLP/gRPC] and https://opentelemetry.io/docs/specs/otlp/#otlphttp[OTLP/HTTP] protocol on the same port as Elastic APM agent requests. For ease of setup, we recommend using OTLP/HTTP when proxying or load balancing requests to the APM Server. |
| 150 | + |
| 151 | +If you use the OTLP/gRPC protocol, requests to the APM Server must use either HTTP/2 over TLS or HTTP/2 Cleartext (H2C). No matter which protocol is used, OTLP/gRPC requests will have the header: `"Content-Type: application/grpc"`. |
| 152 | + |
| 153 | +When using a layer 7 (L7) proxy like AWS ALB, requests must be proxied in a way that ensures requests to the APM Server follow the rules outlined above. For example, with ALB you can create rules to select an alternative backend protocol based on the headers of requests coming into ALB. In this example, you'd select the gRPC protocol when the `"Content-Type: application/grpc"` header exists on a request. |
| 154 | + |
| 155 | +For more information on how to configure an AWS ALB to support gRPC, see this AWS blog post: |
| 156 | +https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/[Application Load Balancer Support for End-to-End HTTP/2 and gRPC]. |
| 157 | + |
| 158 | +For more information on how APM Server services gRPC requests, see |
| 159 | +https://github.com/elastic/apm-server/blob/main/dev_docs/otel.md#muxing-grpc-and-http11[Muxing gRPC and HTTP/1.1]. |
| 160 | + |
| 161 | +[discrete] |
| 162 | +[[apm-open-telemetry-direct-next]] |
| 163 | +== Next steps |
| 164 | + |
| 165 | +* <<apm-open-telemetry-collect-metrics>> |
| 166 | +* Learn about the <<apm-open-telemetry-known-limitations,limitations of this integration>> |
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?v=4&size=40){kind=avatar}
0 commit comments