RCA of 8.19.0 and 9.1.0 APM integration reloader causing unresponsiveness or panic

[carsonip]

This is an issue for root cause analysis of elastic/fleet-server#5170 , to uncover the reason for grayed out "copy endpoint" on ECH cloud admin UI, while reporting APM input as "healthy" in Fleet & agent. As for the fix of the issue itself, it is already done in #17872.

Key questions to be answered:

• did apm server crash on startup for deployments with "copy endpoint" grayed out?
• why is apm server not restarting?

[carsonip]

Observation 1

Logs from apm-server with grayed out "copy endpoint", but no more logs after the last Aggregation.ServiceDestinations.MaxGroups set to 10000 based on 1.0gb of memory.

logs from stuck apm-server

{"log.level":"info","@timestamp":"2025-08-04T15:41:58.096Z","message":"Starting apm-server [9f6812983335fb3d84e0f7100f96a50ff98914b9 built 2025-07-23 17:54:36 +0200 +0200]. Hit CTRL-C to stop it.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":224,"file.name":"beater/server.go","function":"github.com/elastic/apm-server/internal/beater.server.run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:58.096Z","message":"RUM endpoints enabled!","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":86,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-08-04T15:41:58.096Z","message":"CORS related setting `apm-server.rum.allow_origins` allows all origins. Consider more restrictive setting for production use.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":89,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:58.096Z","message":"SSL enabled.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":98,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:58.104Z","message":"no longer blocking ingestion as all precondition checks are now satisfied","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":68,"file.name":"beater/waitready.go","function":"github.com/elastic/apm-server/internal/beater.waitReady"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:58.948Z","message":"BeatV2Manager.unitListen UnitChanged.ID(apm-es-containerhost-elastic-cloud-apm), UnitChanged.Type(modified), UnitChanged.Trigger(2): modified/config_change_triggered","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":507,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.949Z","message":"received input from elastic-agent","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":59,"file.name":"apm-server/main.go","function":"main.init.0.func1"},"ecs.version":"1.6.0","service.name":"apm-server","agent.id":"278c51e5-c395-4a8c-a691-2acaab0e6fd4","agent.version":"9.1.0","unit.id":"elastic-cloud-apm","unit.revision":2,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"using output.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":67,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"config","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"using apm-server.agent.config.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"config","log.origin":{"file.line":73,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"Falling back to elasticsearch output for API Key usage","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"config","log.origin":{"file.line":75,"file.name":"config/auth.go","function":"github.com/elastic/apm-server/internal/beater/config.(*APIKeyAgentAuth).setup"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"Listening on: [::]:8200","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":142,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.listen"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"Stop listening on: 0.0.0.0:8200","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":110,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.(*httpServer).stop"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"stopping apm-server... waiting maximum of 30s for queues to drain","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":197,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run.func1"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"Server stopped","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":243,"file.name":"beater/server.go","function":"github.com/elastic/apm-server/internal/beater.server.run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"stopping aggregator","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"aggregator","log.origin":{"file.line":292,"file.name":"aggregators/aggregator.go","function":"github.com/elastic/apm-aggregation/aggregators.(*Aggregator).Close"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"LSM aggregator stopped","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":194,"file.name":"apm-server/main.go","function":"main.runServerWithProcessors.func1"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.951Z","message":"running final aggregation","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"aggregator","log.origin":{"file.line":304,"file.name":"aggregators/aggregator.go","function":"github.com/elastic/apm-aggregation/aggregators.(*Aggregator).Close"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.952Z","message":"update routine done","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.sourcemap","log.origin":{"file.line":130,"file.name":"sourcemap/metadata_fetcher.go","function":"github.com/elastic/apm-server/internal/sourcemap.(*MetadataESFetcher).startBackgroundSync.func1"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.952Z","message":"loaded input config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"revision":2,"ecs.version":"1.6.0","log.origin":{"file.line":186,"file.name":"beatcmd/reloader.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reloadInputs"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.954Z","message":"MaxConcurrentDecoders set to 102 based on 80 percent of 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":221,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.954Z","message":"Aggregation.MaxServices set to 1000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":228,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.954Z","message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 1000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":235,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.954Z","message":"Aggregation.Transactions.MaxGroups set to 5000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":242,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T15:41:59.954Z","message":"Aggregation.ServiceDestinations.MaxGroups set to 10000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":249,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}

In a successful deployment, it should proceed past Aggregation.ServiceDestinations.MaxGroups set to 10000 based on 1.0gb of memory and the next log line is APM instrumentation is enabled.

logs from successful apm-server

{"log.level":"info","@timestamp":"2025-08-04T14:18:28.139Z","message":"Aggregation.MaxServices set to 1000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":228,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T14:18:28.139Z","message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 1000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":235,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T14:18:28.139Z","message":"Aggregation.Transactions.MaxGroups set to 5000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":242,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T14:18:28.139Z","message":"Aggregation.ServiceDestinations.MaxGroups set to 10000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":249,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T14:18:28.139Z","message":"APM instrumentation is enabled","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.tracing","log.origin":{"file.line":171,"file.name":"instrumentation/instrumentation.go","function":"github.com/elastic/beats/v7/libbeat/instrumentation.initTracer"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T14:18:28.142Z","message":"blocking ingestion until all preconditions are satisfied","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":40,"file.name":"beater/waitready.go","function":"github.com/elastic/apm-server/internal/beater.waitReady"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T14:18:28.142Z","message":"docappender.DocumentBufferSize set to 819 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":828,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.docappenderConfig"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T14:18:28.142Z","message":"docappender.MaxRequests set to 11 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":845,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.docappenderConfig"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T14:18:28.144Z","message":"Path / added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.handler","log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Observation 2

Upon closer examination with @\gigerdo on a ECH deployment container with grayed out "copy endpoint", the apm server process is alive, but it is not listening to 8200, and is stuck in such a state indefinitely.

Observation 3

When enrolling an agent to a policy with APM integration, sometimes apm-server will panic and restart.

logs

{"log.level":"warn","@timestamp":"2025-08-04T16:44:35.270Z","message":"cgroup memory subsystem unavailable","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":1021,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.processMemoryLimit"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:44:35.271Z","message":"no cgroups detected, falling back to total system memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beater","log.origin":{"file.line":1031,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.processMemoryLimit"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:44:35.271Z","message":"MaxConcurrentDecoders set to 2048 based on 80 percent of 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beater","log.origin":{"file.line":221,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:44:35.271Z","message":"Aggregation.MaxServices set to 36567 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beater","log.origin":{"file.line":228,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:44:35.271Z","message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 36567 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":235,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:44:35.271Z","message":"Aggregation.Transactions.MaxGroups set to 182837 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":242,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:44:35.271Z","message":"Aggregation.ServiceDestinations.MaxGroups set to 187837 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":249,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:28.214Z","message":"BeatV2Manager.unitListen UnitChanged.ID(apm-default-28410545-3f3c-4447-a20e-2edbada77ad8), UnitChanged.Type(modified), UnitChanged.Trigger(2): modified/config_change_triggered","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":507,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:29.215Z","message":"received input from elastic-agent","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"agent.id":"92297473-8e55-4554-aec1-617b706c1778","agent.version":"9.1.0","unit.id":"28410545-3f3c-4447-a20e-2edbada77ad8","unit.revision":7,"log.origin":{"file.line":59,"file.name":"apm-server/main.go","function":"main.init.0.func1"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:29.216Z","message":"using output.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"config","log.origin":{"file.line":67,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:29.216Z","message":"using apm-server.agent.config.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":73,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"config","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:29.216Z","message":"Listening on: 127.0.0.1:8200","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":142,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.listen"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"panic: recovered from errgroup.Group: name apm-server already used","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"goroutine 395 [running]:","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"runtime/debug.Stack()","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"runtime/debug/stack.go:26 +0x5e","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"golang.org/x/sync/errgroup.(*Group).add.func1.1()","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"golang.org/x/sync@v0.15.0/errgroup/errgroup.go:117 +0x1e5","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"panic({0x14211c0?, 0xc0006496c0?})","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"runtime/panic.go:792 +0x132","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/elastic-agent-libs/monitoring.panicErr(...)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/elastic-agent-libs@v0.21.0/monitoring/registry.go:287","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/elastic-agent-libs/monitoring.(*Registry).Add(0xc000418dc0, {0x16afd3d?, 0x51?}, {0x1936c40, 0xc00110c600}, 0x37?)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/elastic-agent-libs@v0.21.0/monitoring/registry.go:185 +0xc5","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/elastic-agent-libs/monitoring.(*Registry).NewRegistry(0xc000418dc0, {0x16afd3d, 0xa}, {0x0?, 0x0?, 0x13cb5c0?})","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/elastic-agent-libs@v0.21.0/monitoring/registry.go:94 +0x155","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/apm-server/internal/beater.recordAPMServerConfig(0xc00013b508, 0x0?)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/apm-server/internal/beater/telemetry.go:28 +0x32","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/apm-server/internal/beater.(*Runner).Run(0xc0005ef5e0, {0x19457f0, 0xc000db2370})","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/apm-server/internal/beater/beater.go:263 +0x929","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reload.func1()","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.219Z","message":"github.com/elastic/apm-server/internal/beatcmd/reloader.go:283 +0x3c","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"golang.org/x/sync/errgroup.(*Group).add.func1()","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"golang.org/x/sync@v0.15.0/errgroup/errgroup.go:128 +0x7e","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"created by golang.org/x/sync/errgroup.(*Group).add in goroutine 28","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"golang.org/x/sync@v0.15.0/errgroup/errgroup.go:96 +0x76","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"goroutine 28 [running]:","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"golang.org/x/sync/errgroup.(*Group).Wait(0xc000dac7e0)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"golang.org/x/sync@v0.15.0/errgroup/errgroup.go:70 +0x6d","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reload.func2()","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/apm-server/internal/beatcmd/reloader.go:291 +0x2c","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reload(0xc00011ed00, 0xc000d01380, 0xc000457b00, 0x0)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/apm-server/internal/beatcmd/reloader.go:296 +0x5d6","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reloadInputs(0xc00011ed00, {0xc00011c1d8, 0x3e39d60f43078eee?, 0x0?})","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/apm-server/internal/beatcmd/reloader.go:177 +0x1cd","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/apm-server/internal/beatcmd.reloadableListFunc.Reload(0xc000683140?, {0xc00011c1d8?, 0xc000452090?, 0x30?})","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/apm-server/internal/beatcmd/reloader.go:307 +0x24","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reloadInputs(0xc0004a6120, {0xc00011c060, 0x1, 0x7?})","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250716141617-48de4eb87b27/x-pack/libbeat/management/managerV2.go:836 +0x491","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reload(0xc0004a6120, 0xc000d87e50)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250716141617-48de4eb87b27/x-pack/libbeat/management/managerV2.go:685 +0xb05","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen(0xc0004a6120)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250716141617-48de4eb87b27/x-pack/libbeat/management/managerV2.go:549 +0x585","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 1","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-04T16:45:29.220Z","message":"github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250716141617-48de4eb87b27/x-pack/libbeat/management/managerV2.go:297 +0x1e5","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.232Z","message":"Home path: [/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/components] Config path: [/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/components] Data path: [/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/run/apm-default] Logs path: [/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/components/logs]","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":176,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.232Z","message":"Beat ID: ac57cfec-ede3-4b85-91c7-7210b4697ae6","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":183,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.232Z","message":"Set gc percentage to: 100","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.origin":{"file.line":197,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.233Z","message":"Starting stats endpoint","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"api","log.origin":{"file.line":65,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.233Z","message":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/JQRkhAx9bTDKLV8m3LE2wPv6f9Y0WzTQ.sock (configured: unix:///opt/Elastic/Agent/data/tmp/JQRkhAx9bTDKLV8m3LE2wPv6f9Y0WzTQ.sock)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":67,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.233Z","message":"GOMEMLIMIT is updated","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"package":"github.com/KimMachineGun/automemlimit/memlimit","GOMEMLIMIT":56540263219,"previous":9223372036854776000,"ecs.version":"1.6.0","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.233Z","message":"Beat info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","system_info":{"beat":{"path":{"config":"/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/components","data":"/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/run/apm-default","home":"/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/components","logs":"/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/components/logs"},"type":"apm-server","uuid":"ac57cfec-ede3-4b85-91c7-7210b4697ae6"},"ecs.version":"1.6.0"},"log.logger":"beat","log.origin":{"file.line":861,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.233Z","message":"Build info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","system_info":{"build":{"commit":"9f6812983335fb3d84e0f7100f96a50ff98914b9","time":"2025-07-23T15:54:36.000Z","version":"9.1.0"},"ecs.version":"1.6.0"},"log.logger":"beat","log.origin":{"file.line":869,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.233Z","message":"maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0","log.origin":{"file.line":68,"file.name":"beatcmd/maxprocs.go","function":"github.com/elastic/apm-server/internal/beatcmd.adjustMaxProcs.diffInfof.func2"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.233Z","message":"Go runtime info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beat","log.origin":{"file.line":872,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"service.name":"apm-server","system_info":{"ecs.version":"1.6.0","go":{"arch":"amd64","max_procs":16,"os":"linux","version":"go1.24.5"}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.234Z","message":"Host info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beat","log.origin":{"file.line":876,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"service.name":"apm-server","system_info":{"ecs.version":"1.6.0","host":{"architecture":"x86_64","boot_time":"2025-08-01T15:31:19Z","containerized":false,"ip":["127.0.0.1/8","172.17.0.4/16","::1/128"],"kernel_version":"6.14.0-24-generic","mac":["ba:b3:b3:60:7a:f8"],"name":"73f566464e7b","native_architecture":"x86_64","os":{"codename":"noble","family":"debian","major":24,"minor":4,"name":"Ubuntu","patch":2,"platform":"ubuntu","type":"linux","version":"24.04.2 LTS (Noble Numbat)"},"timezone":"UTC","timezone_offset_sec":0}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.235Z","message":"Process info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","system_info":{"ecs.version":"1.6.0","process":{"capabilities":{"ambient":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"]},"cwd":"/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/run/apm-default","exe":"/opt/Elastic/Agent/data/elastic-agent-9.1.0-2bcf7b/components/apm-server","name":"apm-server","pid":436,"ppid":116,"seccomp":{"mode":"filter","no_new_privs":false},"start_time":"2025-08-04T16:45:39.050Z"}},"log.logger":"beat","log.origin":{"file.line":905,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.236Z","message":"apm-server started.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":432,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.240Z","message":"BeatV2Manager.unitListen UnitChanged.ID(apm-default), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":507,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:39.240Z","message":"BeatV2Manager.unitListen UnitChanged.ID(apm-default-28410545-3f3c-4447-a20e-2edbada77ad8), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":507,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.242Z","message":"loaded output config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0","log.origin":{"file.line":200,"file.name":"beatcmd/reloader.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reloadOutput"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.242Z","message":"received input from elastic-agent","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","agent.version":"9.1.0","unit.id":"28410545-3f3c-4447-a20e-2edbada77ad8","unit.revision":7,"ecs.version":"1.6.0","log.origin":{"file.line":59,"file.name":"apm-server/main.go","function":"main.init.0.func1"},"agent.id":"92297473-8e55-4554-aec1-617b706c1778","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.247Z","message":"using output.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":67,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"config","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.247Z","message":"using apm-server.agent.config.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"config","log.origin":{"file.line":73,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.247Z","message":"Listening on: 127.0.0.1:8200","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":142,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.listen"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.247Z","message":"loaded input config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.origin":{"file.line":186,"file.name":"beatcmd/reloader.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reloadInputs"},"service.name":"apm-server","revision":7,"ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-08-04T16:45:40.248Z","message":"cgroup memory subsystem unavailable","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":1021,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.processMemoryLimit"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.248Z","message":"no cgroups detected, falling back to total system memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":1031,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.processMemoryLimit"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.248Z","message":"MaxConcurrentDecoders set to 2048 based on 80 percent of 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":221,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.248Z","message":"Aggregation.MaxServices set to 36567 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beater","log.origin":{"file.line":228,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.248Z","message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 36567 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":235,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.248Z","message":"Aggregation.Transactions.MaxGroups set to 182837 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beater","log.origin":{"file.line":242,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.248Z","message":"Aggregation.ServiceDestinations.MaxGroups set to 187837 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beater","log.origin":{"file.line":249,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-04T16:45:40.249Z","message":"docappender.DocumentBufferSize set to 29956 based on 36.6gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"log.logger":"beater","log.origin":{"file.line":828,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.docappenderConfig"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}

which show a panic

panic: recovered from errgroup.Group: name apm-server already used
goroutine 395 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:26 +0x5e
golang.org/x/sync/errgroup.(*Group).add.func1.1()
golang.org/x/sync@v0.15.0/errgroup/errgroup.go:117 +0x1e5
panic({0x14211c0?, 0xc0006496c0?})
runtime/panic.go:792 +0x132
github.com/elastic/elastic-agent-libs/monitoring.panicErr(...)
github.com/elastic/elastic-agent-libs@v0.21.0/monitoring/registry.go:287
github.com/elastic/elastic-agent-libs/monitoring.(*Registry).Add(0xc000418dc0, {0x16afd3d?, 0x51?}, {0x1936c40, 0xc00110c600}, 0x37?)
github.com/elastic/elastic-agent-libs@v0.21.0/monitoring/registry.go:185 +0xc5
github.com/elastic/elastic-agent-libs/monitoring.(*Registry).NewRegistry(0xc000418dc0, {0x16afd3d, 0xa}, {0x0?, 0x0?, 0x13cb5c0?})
github.com/elastic/elastic-agent-libs@v0.21.0/monitoring/registry.go:94 +0x155
github.com/elastic/apm-server/internal/beater.recordAPMServerConfig(0xc00013b508, 0x0?)
github.com/elastic/apm-server/internal/beater/telemetry.go:28 +0x32
github.com/elastic/apm-server/internal/beater.(*Runner).Run(0xc0005ef5e0, {0x19457f0, 0xc000db2370})
github.com/elastic/apm-server/internal/beater/beater.go:263 +0x929
github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reload.func1()
github.com/elastic/apm-server/internal/beatcmd/reloader.go:283 +0x3c
golang.org/x/sync/errgroup.(*Group).add.func1()

The question

The question here is, why is apm integration sometimes stuck in a state where it is not listening and not crashing, while the other times they panic and restart immediately?

Hypothesis

apm-server is stuck after giving up listening to port 8200.

Reproducing the stuck state

Here's a test to reproduce a stuck state carsonip@54e95e5

diff

diff --git a/internal/beatcmd/beat_test.go b/internal/beatcmd/beat_test.go
index 732bbd683..945dd200d 100644
--- a/internal/beatcmd/beat_test.go
+++ b/internal/beatcmd/beat_test.go
@@ -576,6 +576,127 @@ func TestRunManager_Reloader(t *testing.T) {
 	}
 }
 
+func TestRunManager_Reloader_panic(t *testing.T) {
+	// This test asserts that panic in runner is propagated immediately.
+	registry := reload.NewRegistry()
+
+	reloader, err := NewReloader(beat.Info{
+		Logger: logptest.NewTestingLogger(t, ""),
+	}, registry, func(p RunnerParams) (Runner, error) {
+		return runnerFunc(func(ctx context.Context) error {
+			revision, err := p.Config.Int("revision", -1)
+			require.NoError(t, err)
+			if revision == 2 {
+				panic("boom!")
+			}
+			return nil
+		}), nil
+	}, nil, nil, nil, beat.NewMonitoring())
+	require.NoError(t, err)
+
+	agentInfo := &proto.AgentInfo{
+		Id:       "elastic-agent-id",
+		Version:  version.VersionWithQualifier(),
+		Snapshot: true,
+	}
+	srv := integration.NewMockServer([]*proto.CheckinExpected{
+		{
+			AgentInfo: agentInfo,
+			Units: []*proto.UnitExpected{
+				{
+					Id:             "output-unit",
+					Type:           proto.UnitType_OUTPUT,
+					ConfigStateIdx: 1,
+					Config: &proto.UnitExpectedConfig{
+						Id:   "default",
+						Type: "elasticsearch",
+						Name: "elasticsearch",
+					},
+					State:    proto.State_HEALTHY,
+					LogLevel: proto.UnitLogLevel_INFO,
+				},
+				{
+					Id:             "input-unit-1",
+					Type:           proto.UnitType_INPUT,
+					ConfigStateIdx: 1,
+					Config: &proto.UnitExpectedConfig{
+						Id:   "elastic-apm",
+						Type: "apm",
+						Name: "Elastic APM",
+						Streams: []*proto.Stream{
+							{
+								Id: "elastic-apm",
+								Source: integration.RequireNewStruct(t, map[string]interface{}{
+									"revision": 1,
+								}),
+							},
+						},
+					},
+					State:    proto.State_HEALTHY,
+					LogLevel: proto.UnitLogLevel_INFO,
+				},
+			},
+			Features:    nil,
+			FeaturesIdx: 1,
+		},
+		{
+			AgentInfo: agentInfo,
+			Units: []*proto.UnitExpected{
+				{
+					Id:             "output-unit",
+					Type:           proto.UnitType_OUTPUT,
+					ConfigStateIdx: 1,
+					State:          proto.State_HEALTHY,
+					LogLevel:       proto.UnitLogLevel_INFO,
+				},
+				{
+					Id:             "elastic-apm",
+					Type:           proto.UnitType_INPUT,
+					ConfigStateIdx: 2,
+					Config: &proto.UnitExpectedConfig{
+						Id:   "elastic-apm",
+						Type: "apm",
+						Name: "Elastic APM",
+						Streams: []*proto.Stream{
+							{
+								Id: "elastic-apm",
+								Source: integration.RequireNewStruct(t, map[string]interface{}{
+									"revision": 2,
+								}),
+							},
+						},
+					},
+					State:    proto.State_HEALTHY,
+					LogLevel: proto.UnitLogLevel_INFO,
+				},
+			},
+			Features:    nil,
+			FeaturesIdx: 1,
+		},
+	},
+		nil,
+		10*time.Millisecond,
+	)
+	require.NoError(t, srv.Start())
+	defer srv.Stop()
+
+	client := client.NewV2(
+		fmt.Sprintf(":%d", srv.Port),
+		"",
+		client.VersionInfo{},
+		client.WithGRPCDialOptions(grpc.WithTransportCredentials(insecure.NewCredentials())))
+	manager, err := xpacklbmanagement.NewV2AgentManagerWithClient(&xpacklbmanagement.Config{
+		Enabled: true,
+	}, registry, client, xpacklbmanagement.WithChangeDebounce(0))
+	require.NoError(t, err)
+
+	err = manager.Start()
+	require.NoError(t, err)
+	defer manager.Stop()
+
+	reloader.Run(context.Background())
+}
+
 func TestRunManager_Reloader_newRunnerError(t *testing.T) {
 	// This test asserts that any errors when creating runner inside reloader, e.g. config parsing error,
 	// will cause the unit to fail.

And the command to test

go test -run=TestRunManager_Reloader_panic -timeout=5s ./...

output

$ go test -run=TestRunManager_Reloader_panic -timeout=5s ./...
?       github.com/elastic/apm-server/cmd/apm-server    [no test files]
ok      github.com/elastic/apm-server/internal/agentcfg 0.006s [no tests to run]
panic: test timed out after 5s
        running tests:
                TestRunManager_Reloader_panic (5s)

goroutine 41 [running]:
testing.(*M).startAlarm.func1()
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/testing/testing.go:2484 +0x394
created by time.goFunc
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/time/sleep.go:215 +0x2d

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0001f8c40, {0x1d97e29?, 0xc00031fb30?}, 0x1e55378)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/testing/testing.go:1859 +0x431
testing.runTests.func1(0xc0001f8c40)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/testing/testing.go:2279 +0x37
testing.tRunner(0xc0001f8c40, 0xc00031fc70)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/testing/testing.go:1792 +0xf4
testing.runTests(0xc0002189d8, {0x2ae85e0, 0x13, 0x13}, {0x2b6c1e0?, 0x7?, 0x2b6a480?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/testing/testing.go:2277 +0x4b4
testing.(*M).Run(0xc0002eb680)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/testing/testing.go:2142 +0x64a
main.main()
        _testmain.go:81 +0x9b

goroutine 8 [chan receive (nil chan)]:
github.com/elastic/apm-server/internal/beatcmd.(*Reloader).Run(0xc0002de820, {0x1feb8f8?, 0x2b8d380?})
        /home/carson/projects/apm-server/internal/beatcmd/reloader.go:135 +0x69
github.com/elastic/apm-server/internal/beatcmd.TestRunManager_Reloader_panic(0xc0001f8e00)
        /home/carson/projects/apm-server/internal/beatcmd/beat_test.go:697 +0xed0
testing.tRunner(0xc0001f8e00, 0x1e55378)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/testing/testing.go:1851 +0x413

goroutine 9 [IO wait]:
internal/poll.runtime_pollWait(0x715f8fe1deb0, 0x72)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0002fc600?, 0x900b637de?, 0x0)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0002fc600)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0002fc600)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc0002031c0)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc0002031c0)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/net/tcpsock.go:380 +0x30
google.golang.org/grpc.(*Server).Serve(0xc0001d2c00, {0x1fe8320, 0xc0002031c0})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:890 +0x473
github.com/elastic/elastic-agent-client/v7/pkg/client/mock.(*StubServerV2).Start.func1()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/mock/stub_serverV2.go:144 +0x34
created by github.com/elastic/elastic-agent-client/v7/pkg/client/mock.(*StubServerV2).Start in goroutine 8
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/mock/stub_serverV2.go:143 +0x159

goroutine 10 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0003803e0, {0x1febb08, 0xc0003820a0})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/grpcsync/callback_serializer.go:88 +0x10c
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 8
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 11 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000380410, {0x1febb08, 0xc0003820f0})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/grpcsync/callback_serializer.go:88 +0x10c
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 8
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 12 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000380440, {0x1febb08, 0xc000382140})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/grpcsync/callback_serializer.go:88 +0x10c
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 8
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 51 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:398 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 23
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:397 +0xe09

goroutine 22 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0xc000482390?)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0x1febb08?)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/sync/waitgroup.go:118 +0x48
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip(0xc0001f9180)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:501 +0x1e9
github.com/elastic/elastic-agent-client/v7/pkg/client.durationOf(0xc000099f80)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:406 +0x31
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startCheckin.func1()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:435 +0xcb
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startCheckin in goroutine 8
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:427 +0x65

goroutine 23 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0xc000442080?)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0x1febb08?)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/sync/waitgroup.go:118 +0x48
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip(0xc0001f9180, 0xc00012e310)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:841 +0x228
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions.func1.1()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:781 +0x1b
github.com/elastic/elastic-agent-client/v7/pkg/client.durationOf(0xc00016c778)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:406 +0x31
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions.func1()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:780 +0xdc
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions in goroutine 8
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:772 +0x8d

goroutine 24 [select]:
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).watchErrChan(0xc0002045a0, {0x1febb08, 0xc000126230})
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250716141617-48de4eb87b27/x-pack/libbeat/management/managerV2.go:463 +0xc9
created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 8
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250716141617-48de4eb87b27/x-pack/libbeat/management/managerV2.go:289 +0x126

goroutine 25 [select]:
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen(0xc0002045a0)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250716141617-48de4eb87b27/x-pack/libbeat/management/managerV2.go:488 +0x209
created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 8
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250716141617-48de4eb87b27/x-pack/libbeat/management/managerV2.go:297 +0x1e5

goroutine 27 [syscall]:
os/signal.signal_recv()
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 25
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/os/signal/signal.go:152 +0x1f

goroutine 52 [select]:
google.golang.org/grpc/internal/transport.(*ClientStream).waitOnHeader(0xc000416140)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/client_stream.go:92 +0x65
google.golang.org/grpc/internal/transport.(*ClientStream).RecvCompress(...)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/client_stream.go:107
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00034a1a0, {0x1cd0d00, 0xc0002fcb80}, 0xc000239ce0?)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:1122 +0x117
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xb64174?)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:976 +0x1f
google.golang.org/grpc.(*clientStream).withRetry(0xc000350000, 0xc0000aceb0, 0xc0000acea0)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:796 +0x13b
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000350000, {0x1cd0d00?, 0xc0002fcb80?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:975 +0x155
github.com/elastic/elastic-agent-client/v7/pkg/proto.(*elasticAgentActionsClient).Recv(0xc000418170)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/proto/elastic-agent-client_grpc.pb.go:140 +0x46
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionsReader(0xc0001f9180, {0x1ff57b8, 0xc000418170}, 0xc00012e310)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:848 +0x35
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip.func1()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:829 +0x7a
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip in goroutine 23
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:826 +0x172

goroutine 15 [IO wait]:
internal/poll.runtime_pollWait(0x715f8fe1dd98, 0x72)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0002fcb00?, 0xc0003a0000?, 0x0)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002fcb00, {0xc0003a0000, 0x8000, 0x8000})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc0002fcb00, {0xc0003a0000?, 0xc0000c4600?, 0xb64174?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00009aba8, {0xc0003a0000?, 0xb5bae0?, 0x1c0004ba000?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/net/net.go:194 +0x45
bufio.(*Reader).Read(0xc0000c5aa0, {0xc000000200, 0x9, 0xbc5359?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/bufio/bufio.go:245 +0x197
io.ReadAtLeast({0x1fda480, 0xc0000c5aa0}, {0xc000000200, 0x9, 0x9}, 0x9)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/io/io.go:335 +0x91
io.ReadFull(...)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000000200, 0x9, 0x49ee65c01620?}, {0x1fda480?, 0xc0000c5aa0?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/x/net@v0.41.0/http2/frame.go:242 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000001c0)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/x/net@v0.41.0/http2/frame.go:506 +0x7d
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00024cd88, 0xc0002db110)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_client.go:1662 +0x1d2
created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 21
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_client.go:414 +0x1e73

goroutine 38 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:398 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 22
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:397 +0xe09

goroutine 50 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000203340, 0x1)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/controlbuf.go:412 +0x108
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000506080)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/controlbuf.go:575 +0x78
google.golang.org/grpc/internal/transport.NewHTTP2Client.func6()
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_client.go:472 +0xd2
created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 21
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_client.go:470 +0x2452

goroutine 35 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0004b2000, 0x1)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/controlbuf.go:412 +0x108
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000354080)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/controlbuf.go:575 +0x78
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_server.go:337 +0xde
created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 34
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_server.go:335 +0x189b

goroutine 36 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0004a8000)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_server.go:1198 +0x1ec
created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 34
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_server.go:358 +0x18dd

goroutine 37 [IO wait]:
internal/poll.runtime_pollWait(0x715f8fe1dc80, 0x72)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000480000?, 0xc00048e000?, 0x0)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000480000, {0xc00048e000, 0x8000, 0x8000})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc000480000, {0xc00048e000?, 0xc0006a9dc0?, 0xc00044dba8?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000484000, {0xc00048e000?, 0x715f406ba368?, 0x715f8fe96108?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/net/net.go:194 +0x45
bufio.(*Reader).Read(0xc00048c000, {0xc0004a0040, 0x9, 0x1c23380?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/bufio/bufio.go:245 +0x197
io.ReadAtLeast({0x1fda480, 0xc00048c000}, {0xc0004a0040, 0x9, 0x9}, 0x9)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/io/io.go:335 +0x91
io.ReadFull(...)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.5.linux-amd64/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0004a0040, 0x9, 0xbcbe2c?}, {0x1fda480?, 0xc00048c000?})
        /home/carson/apm-server-gopath/pkg/mod/golang.org/x/net@v0.41.0/http2/frame.go:242 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0004a0000)
        /home/carson/apm-server-gopath/pkg/mod/golang.org/x/net@v0.41.0/http2/frame.go:506 +0x7d
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0004a8000, {0x1febad0, 0xc000482210}, 0xc000482240)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/http2_server.go:658 +0x105
google.golang.org/grpc.(*Server).serveStreams(0xc0001d2c00, {0x1feb8f8?, 0x2b8d380?}, {0x1fec570, 0xc0004a8000}, {0x1ff5970?, 0xc000484000?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1029 +0x396
google.golang.org/grpc.(*Server).handleRawConn.func1()
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:964 +0x56
created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 34
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:963 +0x1cb

goroutine 53 [select]:
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionsWriter(0xc0001f9180, {0x1ff57b8, 0xc000418170}, 0xc00012e310, 0xc00042a2a0)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:885 +0x17c
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip.func2()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:837 +0x85
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip in goroutine 23
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:834 +0x21e

goroutine 39 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readMessageHeaderClient(0xc0004960f0, {0xc000482310, 0x5, 0x5})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:197 +0xa7
google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadMessageHeader(0xc0004960f0, {0xc000482310?, 0xc00024cd88?, 0xc0004960a0?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:143 +0x45
google.golang.org/grpc/internal/transport.(*transportReader).ReadMessageHeader(0xc0004aa0a0, {0xc000482310?, 0x1ffc3e8?, 0xc0004130a0?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:417 +0x25
google.golang.org/grpc/internal/transport.(*Stream).ReadMessageHeader(0xc0004bc000, {0xc000482310, 0x5, 0x5})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:360 +0x9e
google.golang.org/grpc.(*parser).recvMsg(0xc000482300, 0x400000)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:690 +0x3b
google.golang.org/grpc.recvAndDecompress(0xc000482300, {0x1fdcb80, 0xc0004960a0}, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0}, 0x0)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:850 +0x97
google.golang.org/grpc.recv(0xc00036dbd0?, {0x715f480220d0, 0x2b8d380}, {0x1fdcb80?, 0xc0004960a0?}, {0x0?, 0x0?}, {0x1cd1060, 0xc00033a540}, 0x400000, ...)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:933 +0xab
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00042c0d0, {0x1cd1060, 0xc00033a540}, 0x0?)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:1136 +0x2e5
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x0?)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:976 +0x1f
google.golang.org/grpc.(*clientStream).withRetry(0xc000430000, 0xc00036de50, 0xc00036de40)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:780 +0x3b0
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000430000, {0x1cd1060?, 0xc00033a540?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:975 +0x155
github.com/elastic/elastic-agent-client/v7/pkg/proto.(*elasticAgentCheckinV2Client).Recv(0xc00048a0f0)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/proto/elastic-agent-client_grpc.pb.go:109 +0x46
github.com/elastic/elastic-agent-client/v7/pkg/client/chunk.RecvExpected({0x715f480220f8, 0xc00048a0f0})
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/chunk/expected.go:108 +0x47
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip.func1()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:484 +0xd4
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip in goroutine 22
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:480 +0x14f

goroutine 40 [select]:
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinWriter(0xc0001f9180, {0x1ff5810, 0xc00048a0f0}, 0xc0004a6380)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:518 +0x118
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip.func2()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:496 +0x65
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip in goroutine 22
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/client_v2.go:494 +0x1df

goroutine 82 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readMessageHeader(0xc00068a0f0, {0xc000502160, 0x5, 0x5})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:176 +0x9e
google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadMessageHeader(0xc00068a0f0, {0xc000502160?, 0xc0004a8000?, 0xc0006aa060?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:145 +0x72
google.golang.org/grpc/internal/transport.(*transportReader).ReadMessageHeader(0xc0006c6040, {0xc000502160?, 0x1ffc3e8?, 0xc000506100?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:417 +0x25
google.golang.org/grpc/internal/transport.(*Stream).ReadMessageHeader(0xc0006c8000, {0xc000502160, 0x5, 0x5})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:360 +0x9e
google.golang.org/grpc.(*parser).recvMsg(0xc000502150, 0x400000)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:690 +0x3b
google.golang.org/grpc.recvAndDecompress(0xc000502150, {0x1fdcae0, 0xc0006aa060}, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0}, 0x1)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:850 +0x97
google.golang.org/grpc.recv(0x2b8f4c0?, {0x715f480220d0, 0x2b8d380}, {0x1fdcae0?, 0xc0006aa060?}, {0x0?, 0x0?}, {0x1cb8ac0, 0xc00013a200}, 0x400000, ...)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:933 +0xab
google.golang.org/grpc.(*serverStream).RecvMsg(0xc00052e000, {0x1cb8ac0, 0xc00013a200})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:1769 +0x192
github.com/elastic/elastic-agent-client/v7/pkg/proto.(*elasticAgentActionsServer).Recv(0xc000500050)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/proto/elastic-agent-client_grpc.pb.go:301 +0x46
github.com/elastic/elastic-agent-client/v7/pkg/client/mock.(*StubServerV2).Actions(0xc0002fc580, {0x1ff58c0, 0xc000500050})
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/mock/stub_serverV2.go:228 +0x13c
github.com/elastic/elastic-agent-client/v7/pkg/proto._ElasticAgent_Actions_Handler({0x1d165c0?, 0xc0002fc580}, {0x1ff1110, 0xc00052e000})
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/proto/elastic-agent-client_grpc.pb.go:282 +0xd8
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001d2c00, {0x1febad0, 0xc0005020f0}, 0xc0006aa060, 0xc000201cb0, 0x2ae5b40, 0x0)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1695 +0x1252
google.golang.org/grpc.(*Server).handleStream(0xc0001d2c00, {0x1fec570, 0xc0004a8000}, 0xc0006aa060)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1819 +0xb47
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1035 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 37
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1046 +0x11d

goroutine 83 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readMessageHeader(0xc00068a1e0, {0xc0005900a0, 0x5, 0x5})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:176 +0x9e
google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadMessageHeader(0xc00068a1e0, {0xc0005900a0?, 0xc0004a8000?, 0xc0006aa0c0?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:145 +0x72
google.golang.org/grpc/internal/transport.(*transportReader).ReadMessageHeader(0xc0006c60a0, {0xc0005900a0?, 0xb5bc45?, 0xc00001d5b0?})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:417 +0x25
google.golang.org/grpc/internal/transport.(*Stream).ReadMessageHeader(0xc0006c8120, {0xc0005900a0, 0x5, 0x5})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/internal/transport/transport.go:360 +0x9e
google.golang.org/grpc.(*parser).recvMsg(0xc000590090, 0x400000)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:690 +0x3b
google.golang.org/grpc.recvAndDecompress(0xc000590090, {0x1fdcae0, 0xc0006aa0c0}, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0}, 0x1)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:850 +0x97
google.golang.org/grpc.recv(0x15d6ca?, {0x715f480220d0, 0x2b8d380}, {0x1fdcae0?, 0xc0006aa0c0?}, {0x0?, 0x0?}, {0x1cdc400, 0xc000429710}, 0x400000, ...)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/rpc_util.go:933 +0xab
google.golang.org/grpc.(*serverStream).RecvMsg(0xc00059e000, {0x1cdc400, 0xc000429710})
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/stream.go:1769 +0x192
github.com/elastic/elastic-agent-client/v7/pkg/proto.(*elasticAgentCheckinV2Server).Recv(0xc0005a0010)
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/proto/elastic-agent-client_grpc.pb.go:275 +0x46
github.com/elastic/elastic-agent-client/v7/pkg/client/chunk.RecvObserved({0x715f406f9108, 0xc0005a0010})
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/chunk/observed.go:110 +0x47
github.com/elastic/elastic-agent-client/v7/pkg/client/mock.(*StubServerV2).CheckinV2(0xc0002fc580, {0x1ff5918, 0xc0005a0010})
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/mock/stub_serverV2.go:180 +0x56
github.com/elastic/elastic-agent-client/v7/pkg/proto._ElasticAgent_CheckinV2_Handler({0x1d165c0?, 0xc0002fc580}, {0x1ff1110, 0xc00059e000})
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/proto/elastic-agent-client_grpc.pb.go:256 +0xd8
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001d2c00, {0x1febad0, 0xc000590030}, 0xc0006aa0c0, 0xc000201cb0, 0x2ae5b20, 0x0)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1695 +0x1252
google.golang.org/grpc.(*Server).handleStream(0xc0001d2c00, {0x1fec570, 0xc0004a8000}, 0xc0006aa0c0)
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1819 +0xb47
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1035 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 37
        /home/carson/apm-server-gopath/pkg/mod/google.golang.org/grpc@v1.73.0/server.go:1046 +0x11d

goroutine 98 [select]:
github.com/elastic/elastic-agent-client/v7/pkg/client/mock.(*StubServerV2).Actions.func1()
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/mock/stub_serverV2.go:202 +0xb9
created by github.com/elastic/elastic-agent-client/v7/pkg/client/mock.(*StubServerV2).Actions in goroutine 82
        /home/carson/apm-server-gopath/pkg/mod/github.com/elastic/elastic-agent-client/v7@v7.17.2/pkg/client/mock/stub_serverV2.go:200 +0xfe
FAIL    github.com/elastic/apm-server/internal/beatcmd  5.025s
ok      github.com/elastic/apm-server/internal/beater   0.010s [no tests to run]
ok      github.com/elastic/apm-server/internal/beater/api       0.008s [no tests to run]
ok      github.com/elastic/apm-server/internal/beater/api/config/agent  0.005s [no tests to run]
ok      github.com/elastic/apm-server/internal/beater/api/intake        0.005s [no tests to run]
ok      github.com/elastic/apm-server/internal/beater/api/root  0.004s [no tests to run]
ok      github.com/elastic/apm-server/internal/beater/auth      0.006s [no tests to run]
?       github.com/elastic/apm-server/internal/beater/beatertest        [no test files]
ok      github.com/elastic/apm-server/internal/beater/config    0.005s [no tests to run]
?       github.com/elastic/apm-server/internal/beater/headers   [no test files]
ok      github.com/elastic/apm-server/internal/beater/interceptors      0.009s [no tests to run]
ok      github.com/elastic/apm-server/internal/beater/middleware        0.009s [no tests to run]
?       github.com/elastic/apm-server/internal/beater/monitoringtest    [no test files]
ok      github.com/elastic/apm-server/internal/beater/otlp      0.011s [no tests to run]
ok      github.com/elastic/apm-server/internal/beater/ratelimit 0.004s [no tests to run]
ok      github.com/elastic/apm-server/internal/beater/request   0.006s [no tests to run]
ok      github.com/elastic/apm-server/internal/elasticsearch    0.009s [no tests to run]
?       github.com/elastic/apm-server/internal/fips140  [no test files]
ok      github.com/elastic/apm-server/internal/kibana   0.005s [no tests to run]
ok      github.com/elastic/apm-server/internal/logs     0.004s [no tests to run]
ok      github.com/elastic/apm-server/internal/model/modelprocessor     0.009s [no tests to run]
ok      github.com/elastic/apm-server/internal/netutil  0.007s [no tests to run]
ok      github.com/elastic/apm-server/internal/publish  0.038s [no tests to run]
ok      github.com/elastic/apm-server/internal/r8       0.013s [no tests to run]
ok      github.com/elastic/apm-server/internal/sourcemap        0.025s [no tests to run]
ok      github.com/elastic/apm-server/internal/telemetry        0.012s [no tests to run]
?       github.com/elastic/apm-server/internal/version  [no test files]
ok      github.com/elastic/apm-server/x-pack/apm-server 0.013s [no tests to run]
?       github.com/elastic/apm-server/x-pack/apm-server/aggregation     [no test files]
ok      github.com/elastic/apm-server/x-pack/apm-server/sampling        0.008s [no tests to run]
ok      github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage   0.005s [no tests to run]
ok      github.com/elastic/apm-server/x-pack/apm-server/sampling/pubsub 0.005s [no tests to run]
?       github.com/elastic/apm-server/x-pack/apm-server/sampling/pubsub/pubsubtest      [no test files]
FAIL

Observation 4

From the above test, apparently the panic in runner is not propagated immediately.

Observation 5

From observation 3, the panic message contains a "recovered from errgroup.Group". Combining that with observation 4, it appears that errgroup.Group is messing with panics.

apm-server 9.1 uses golang.org/x/errgroup 0.15.0. The g.Go implementation contains a defer that catches panics and recovers from it, and only propagate it on g.Wait.
https://github.com/golang/sync/blob/v0.15.0/errgroup/errgroup.go#L99-L126

		defer func() {
			if normalReturn {
				return
			}
			v := recover()
			g.mu.Lock()
			defer g.mu.Unlock()
			if !g.abnormal {
				if g.cancel != nil {
					g.cancel(g.err)
				}
				g.abnormal = true
			}
			if v != nil && g.panicValue == nil {
				switch v := v.(type) {
				case error:
					g.panicValue = PanicError{
						Recovered: v,
						Stack:     debug.Stack(),
					}
				default:
					g.panicValue = PanicValue{
						Recovered: v,
						Stack:     debug.Stack(),
					}
				}
			}
		}()

Putting this back into the context of apm-server Reloader,

apm-server/internal/beatcmd/reloader.go

Lines 280 to 301 in fc11000

	var g errgroup.Group
	ctx, cancel := context.WithCancel(context.Background())
	g.Go(func() error {
	if err := newRunner.Run(ctx); err != nil && !errors.Is(err, context.Canceled) {
	r.logger.With(logp.Error(err)).Error("runner returned with an error")
	return err
	}
	return nil
	})
	stopRunner := func() error {
	cancel()
	return g.Wait()
	}
	// Stop any existing runner.
	if r.runner != nil {
	_ = r.stopRunner() // logged above
	}
	r.runner = newRunner
	r.stopRunner = stopRunner
	return nil

when function in g.Go panics, everything stops and there is no log, but panic is not surfaced. This explains observation 1 and 2.

Reloader.reload only calls g.Wait in stopRunner, which is when apm-server receives a signal to stop, or an input from EA to hot reload a new config. Practically, panic only shows up on next input from EA. This explains observation 3.

Putting observations together

apm-server Reloader uses errgroup. errgroup implementation delays propagation of panics.

Combining this with the panic caused by stateRegistry.NewRegistry, which only panics on the 2nd call (code and code), here's a timeline of what will happen with respect to number of times apm-server receive input from EA.

• fresh start (no reload, initial config loaded from EA): stateRegistry.NewRegistry does not panic, apm-server loads the config and runs as usual
• 1st reload: stateRegistry.NewRegistry panics, causing errgroup goroutine to panic, but the panic is stored and not propagated. apm-server closes listener and stop emitting logs.
• 2nd reload: calls apm-server Reloader stopRunner, which in turn calls g.Wait from the last reload, which causes the panic to surface. Crashes the apm-server, and EA restarts apm-server process.

Conclusion

The merged PR #17872 is an effective fix to the root cause, as it eliminates the panics in the first place.

However, for the handling of panics, the existing use of errgroup is worrying as it also means any future panic-causing regressions will be as hard to debug as this one. Improvement proposals to be discussed in a follow-up reply.

[carsonip]

Action items to make future panics easier to detect

1. Hand coded fix

Just by replacing errgroup.Go, the panic will surface right away when it happens

diff --git a/internal/beatcmd/reloader.go b/internal/beatcmd/reloader.go
index 08763b5d7..5bfd8af63 100644
--- a/internal/beatcmd/reloader.go
+++ b/internal/beatcmd/reloader.go
@@ -26,7 +26,6 @@ import (
 	"go.elastic.co/apm/module/apmotel/v2"
 	"go.opentelemetry.io/otel/metric"
 	"go.opentelemetry.io/otel/trace"
-	"golang.org/x/sync/errgroup"
 
 	"github.com/elastic/beats/v7/libbeat/beat"
 	"github.com/elastic/beats/v7/libbeat/cfgfile"
@@ -277,18 +276,19 @@ func (r *Reloader) reload(inputConfig, outputConfig, apmTracingConfig *config.C)
 	}
 
 	// Start the new runner.
-	var g errgroup.Group
+	done := make(chan error, 1)
 	ctx, cancel := context.WithCancel(context.Background())
-	g.Go(func() error {
+	go func() {
 		if err := newRunner.Run(ctx); err != nil && !errors.Is(err, context.Canceled) {
 			r.logger.With(logp.Error(err)).Error("runner returned with an error")
-			return err
+			done <- err
 		}
-		return nil
-	})
+		close(done)
+	}()
 	stopRunner := func() error {
 		cancel()
-		return g.Wait()
+		err := <-done
+		return err
 	}
 
 	// Stop any existing runner.

2. Bump golang.org/x/sync

Apparently, errgroup maintainers realized that being too smart around panic handling is bad. In 0.16.0, they removed the recover() in defer: https://github.com/golang/sync/blob/v0.16.0/errgroup/errgroup.go#L79

I have validated that bumping golang.org/x/sync from v0.15.0 to v0.16.0 (diff) causes the new TestRunManager_Reloader_panic in carsonip@54e95e5 to fail right away.

Obviously (2) is better than (1) to surface the panic, as the difficulty to debug is introduced by errgroup in the first place. However, I still haven't figured out how to incorporate the new test TestRunManager_Reloader_panic into the codebase, as any panic introduced in a child goroutine will crash the test immediately, as it will not go through parent goroutine defer. i.e. we cannot

	assert.Panics(t, func() {
		reloader.Run(context.Background())
	})

because reloader.Run does not panic itself, but it has a child goroutine that panics. This needs a bit more work.