TBS: Expose TTL config via integration policy

[carsonip]

Tail based sampling TTL sampling.tail.ttl config is only configurable via apm-server.yml, and is not available to fleet-managed apm-server. TTL is a useful knob to adjust TBS storage usage if storage limit is reached and sampling.tail.storage_limit cannot be further increased for any reason.

Add TTL config to integration policy, APM UI, and docs.

Files requiring changes (and possibly more):

• https://github.com/elastic/integrations/blob/c2ceb26e216b88ec373e599f9c38948594a6b570/packages/apm/agent/input/template.yml.hbs#L71
• https://github.com/elastic/kibana/blob/c5a00472cba967608fc6ba8b47eb7d3deee94780/x-pack/plugins/observability_solution/apm/public/components/fleet_integration/apm_policy_form/settings_definition/tail_sampling_settings.tsx#L99
• https://github.com/elastic/observability-docs/blob/e1279a3edabc9ec2d90e5f041bbcc599bddbec66/docs/en/observability/apm/configure/sampling.asciidoc#L65

[raultorrecilla]

Input from @carsonip for this task:
The task will be about:

1. changing integration package (we can do it very quickly)
2. changing APM integration UI (we may do it, but I think it's better for #obs-ux-infra_services to do it)
3. Updating some translations files for the UI (need to ask #obs-ux-infra_services who's responsible for that).

Moving it to it-109 before 9.1 FF.

[carsonip]

changing integration package (we can do it very quickly)
changing APM integration UI (we may do it, but I think it's better for #obs-ux-infra_services to do it)
Updating some translations files for the UI (need to ask #obs-ux-infra_services who's responsible for that).

I've discussed with UI team internally and apparently the UI will fetch the policy, display configs that it supports, then merge it back to the fetched policy to save it. i.e. If there's a policy config that isn't supported in the UI, there's no side effect on loading and saving them in the UI.

Therefore, we can decouple the integration package work and it is not necessary to release it with the UI at the same time. UI support is preferred but it will not be a blocker.

I'll still target 9.1 for the change in integration package.

Q:

• Should this be backported to 8.19? It will be a knob to adjust storage usage but a rather ineffective one due to 8.x issue TBS: Expired entries stay much longer than TTL and consume disk space #15121

[carsonip]

• docs PR: apm: Document sampling.tail.ttl config docs-content#1269
• 8.x docs PR: apm: Document apm-server.sampling.tail.ttl observability-docs#4914
• apm integration policy UI issue: [APM] Support sampling ttl configuration in apm integration policy kibana#219303
• integration package PR: apm: Add integration policy variable tail_sampling_ttl to configure apm-server.sampling.tail.ttl integrations#13348
  • 8.19 backport: apm: Backport TBS ttl and discard_on_write_failure to 8.19 integrations#14029
• apm-server.yml PR: Add sampling.tail.ttl to sample yml #16579