networkobservability_tcp_connection_remote only emits address=AllIPs instead of per-remote address labels

[abdullazin]

Summary

networkobservability_tcp_connection_remote is documented as a per-remote IP/port metric, but the current implementation only emits a single aggregated series with address="AllIPs".

We are specifically interested in having this supported in the lighter Hubble-based Retina deployment as well, since that is the mode we are using in production.

Expected behavior

The metric description and docs suggest this should expose one series per active remote address/port pair, for example:

• networkobservability_tcp_connection_remote{address="10.0.0.1",port="443"}
• networkobservability_tcp_connection_remote{address="10.0.0.2",port="5432"}

From our perspective, this would be especially valuable if available in the Hubble-based Retina deployment, not just in a heavier/full Retina setup.

Actual behavior

The current code sums all valid remote addresses into a single total and exports only:

• networkobservability_tcp_connection_remote{address="AllIPs"}

Relevant code:

• retina/pkg/plugin/linuxutil/netstat_stats_linux.go

  Lines 236 to 245 in 577d917

  	totalCount := 0
  	for remoteAddr, v := range nr.connStats.TcpSockets.socketByRemoteAddr {
  	// only count valid remote addresses
  	if _, err := netip.ParseAddrPort(remoteAddr); err != nil {
  	nr.l.Error("failed to parse remote address", zap.String("remoteAddr", remoteAddr), zap.Error(err))
  	continue
  	}
  	totalCount += v
  	}
  	metrics.TCPConnectionRemoteGauge.WithLabelValues(addrDefaultTCPRemote).Set(float64(totalCount))

The constant is also defined here:

• retina/pkg/plugin/linuxutil/netstat_stats_linux.go

  Lines 19 to 23 in 577d917

  	const (
  	pathNetNetstat = "/proc/net/netstat"
  	pathNetSnmp = "/proc/net/snmp"
  	addrDefaultTCPRemote = "AllIPs"
  	)

Why this looks incomplete

The implementation already builds socketByRemoteAddr in processSocks(), but updateMetrics() only validates and sums those entries instead of exporting them individually.

It seems likely the missing piece is handling series cleanup across polls, i.e. setting previously seen remote address/port label combinations to 0 or otherwise removing stale series when they disappear from the current snapshot.

Impact

This makes the metric much less useful for diagnosing network issues. Dashboards that group by(address) end up with a single AllIPs series instead of a breakdown by remote endpoint.

In our case, this is affecting a Hubble-based Retina deployment where we want a lightweight but still actionable TCP remote-endpoint diagnostic signal.

Docs mismatch

These docs describe the metric as per remote IP/port:

• https://github.com/microsoft/retina/blob/main/docs/03-Metrics/02-hubble_metrics.md
• https://github.com/microsoft/retina/blob/main/docs/03-Metrics/modes/basic.md

Suggested fix

Either:

1. Implement per-remote address/port series export for networkobservability_tcp_connection_remote, including stale-series handling between polls

or

2. Update the docs/metric description to reflect that the metric is intentionally aggregated into AllIPs

A real per-remote series would be much more useful if cardinality is acceptable or can be gated/configured, and we would especially like to see that supported in the Hubble-based Retina deployment path.