Report native CMN mesh (uncore) metrics for Phoenix (#605)

Summary:

**Problem**
The Neoverse V3 perf report exposed a single CMN memory metric ("CMN Memory
Read Bandwidth") derived from `hns_mc_reqs_local_all`. That name was misleading
(the event counts read+write) and, more importantly, the metric was incomplete:
it ignored remote (cross-chiplet) memory traffic and surfaced none of the other
CMN mesh telemetry the hardware exposes. An earlier attempt to add DRAM
read/write bandwidth via the external Arm cmn-tools repo (port-filtered XP
watchpoints, cloned at install time) produced unusable numbers on Phoenix —
write bandwidth reported as 0 and mesh bandwidth of ~680 GB/s — and added
network/`/dev/mem`/kernel-module fragility to the install path.

**Why**
Root cause of the cmn-tools garbage: Meta fleet telemetry (dynolog) holds ~1.3k
CMN perf_event fds continuously, starving the CMN DTC's small hardware counter
budget. cmn-tools' watchpoint programming collides with the arm-cmn driver under
that contention. Named perf events do not — perf multiplexes and scales them, so
steady-state values stay accurate. Separately, the Cypress HN-S MC-request event
genuinely has no read/write split (confirmed against the event set and the
Phoenix SoC Architecture Specification, Table 15-4), so a DRAM byte-level
read/write split is not observable from HN-S counters at all.

**Fix**
Drop the cmn-tools apparatus entirely (install_platform_tools.sh,
cmn_mesh_discovery.sh, cmn_topdown_runner.sh, the install.py / topdown.py hooks,
and the log-parsing path in the report generator). Instead report native CMN PMU
metrics mapped to the "HNF Effectiveness" group in the Phoenix SoC Architecture
Specification, Table 15-4:

- **CMN Memory Bandwidth (MBps)** — read+write total, now summing local **and**
  remote MC requests across both chiplets (each 64B request counted once at its
  home HN-S, so no double-counting).
- **CMN Local / Remote Memory Bandwidth (MBps)** — NUMA locality split.
- **CMN Memory Read Mix %** — read share via HN-S QoS PoCQ occupancy (the only
  read/write proxy the HN-S exposes).
- **CMN MC Retry %** — memory-controller backpressure (Table 15-4 mc_retry).
- **CMN Snoop Filter Hit Rate %** — coherence-directory effectiveness (sf_hit_ratio).
- **CMN Mesh Frequency (GHz)** — from dtc_cycles, for DVFS-throttling visibility.

The collect script keeps the CMN event group intentionally lean because of the
dynolog counter contention described above.

Differential Revision: D102244354

Author: charles-typ

perfutils/collect_neoversev3_perf_counters.sh

@@ -128,17 +128,38 @@ L2_CHI_BUSY='r198,r199,r19A,r19B,r19C'
 # sve_pred_full_spec (0x8076), sve_pred_partial_spec (0x8077)
 SVE_PRED='r8074,r8075,r8076,r8077'
 
-### CMN-Cypress Uncore PMU Events (SLC / System Level Cache) ---------
+### CMN-Cypress Uncore PMU Events (mesh / SLC / memory) ---------
 ### Auto-discover arm_cmn_N devices (one per chiplet on NV3).
-### HN-S (Home Node with SLC) events provide true SLC miss rate,
-### which is not observable from the core PMU on Neoverse V3.
+### These map to the "HNF Effectiveness" group in the Phoenix SoC
+### Architecture Specification, Table 15-4 (CMN Performance Events):
+###   mc_requests    -> hns_mc_reqs_{local,remote}_all (64B DRAM requests)
+###   mc_retry       -> hns_mc_retries_local
+###   pocq_occupancy -> hns_qos_pocq_occupancy_{read,write}
+###   slc/sf_hit     -> hns_slc_sf_cache_access_all, hns_cache_miss_all,
+###                     hns_cache_fill_all, hns_sf_hit_all
+### plus dtc_cycles for the mesh clock frequency.
+###
+### NOTE ON MULTIPLEXING: the CMN DTC has a small number of hardware counters
+### and Meta fleet telemetry (dynolog) holds several of them continuously, so
+### this event set is kept intentionally lean. perf multiplexes and scales the
+### counts; keeping the group small minimizes scaling error. Read/write DRAM
+### *byte* split is not observable from HN-S events on Cypress (it requires XP
+### watchpoint programming, which collides with the arm-cmn driver), so memory
+### bandwidth is reported as a read+write total with a PoCQ-occupancy-based
+### read/write mix proxy.
 CMN_SLC_EVENTS=""
 for cmn_dev in $(find /sys/bus/event_source/devices/ -maxdepth 1 -name 'arm_cmn_*' -printf '%f\n' 2>/dev/null | sort); do
     CMN_SLC_EVENTS+="${cmn_dev}/hns_slc_sf_cache_access_all/,"
     CMN_SLC_EVENTS+="${cmn_dev}/hns_cache_miss_all/,"
     CMN_SLC_EVENTS+="${cmn_dev}/hns_cache_fill_all/,"
+    CMN_SLC_EVENTS+="${cmn_dev}/hns_sf_hit_all/,"
     CMN_SLC_EVENTS+="${cmn_dev}/hns_mc_reqs_local_all/,"
+    CMN_SLC_EVENTS+="${cmn_dev}/hns_mc_reqs_remote_all/,"
+    CMN_SLC_EVENTS+="${cmn_dev}/hns_mc_retries_local/,"
     CMN_SLC_EVENTS+="${cmn_dev}/hns_pocq_reqs_recvd_all/,"
+    CMN_SLC_EVENTS+="${cmn_dev}/hns_qos_pocq_occupancy_read/,"
+    CMN_SLC_EVENTS+="${cmn_dev}/hns_qos_pocq_occupancy_write/,"
+    CMN_SLC_EVENTS+="${cmn_dev}/dtc_cycles/,"
 done
 CMN_SLC_EVENTS="${CMN_SLC_EVENTS%,}"
 
@@ -152,7 +173,7 @@ CPU_GROUP_MUX="${INSTRUCTIONS_RATE},${L1_DCACHE_MISSES},${L1_ICACHE_MISSES},${L2
 
 PERF_PID=
 wrapup() {
-  kill -INT "$PERF_PID"
+  kill -INT "$PERF_PID" 2>/dev/null
 }
 
 trap wrapup SIGINT SIGTERM
@@ -179,6 +200,7 @@ collect_counters() {
   if [[ -n "$CMN_SLC_EVENTS" ]]; then
     events+=" -e ${CMN_SLC_EVENTS}"
   fi
+
   if [[ -n "$outfile" ]]; then
     perf_stat "$events" "$interval_ms" > "$outfile"
   else

perfutils/generate_arm_neoversev3_perf_report.py

@@ -788,28 +788,176 @@ def dispatch_stall_mcq(grouped_df):
 
 
 # ===========================================================================
-# SVE predication effectiveness (V3-specific)
+# CMN mesh (uncore) metrics
+#
+# CMN-Cypress exposes one arm_cmn_N PMU per chiplet. The HN-S (Home Node with
+# SLC) events below map to the "HNF Effectiveness" group in the Phoenix SoC
+# Architecture Specification, Table 15-4 (CMN Performance Events):
+#   mc_requests   -> hns_mc_reqs_{local,remote}_all
+#   mc_retry      -> hns_mc_retries_{local,remote}
+#   pocq_occupancy-> hns_qos_pocq_occupancy_{read,write,all}
+#   sf_hit_ratio  -> hns_sf_hit_all / hns_slc_sf_cache_access_all
+#
+# Each hns_mc_reqs_* event counts a 64B cache-line request to the memory
+# controller and is counted exactly once at the request's home HN-S, so
+# summing local + remote across both chiplets yields total DRAM traffic with
+# no double-counting. The event does NOT distinguish reads from writes at the
+# HN-S (confirmed against the Cypress event set), which is why the memory
+# bandwidth metric is reported as a read+write total.
 # ===========================================================================
 
 
 @skip_if_missing
-def cmn_mem_read_bw_MBps(grouped_df):
-    """Memory read bandwidth from CMN MC request counters.
+def cmn_mem_bw_MBps(grouped_df):
+    """Total DRAM bandwidth (read+write) from CMN MC request counters.
 
-    Each hns_mc_reqs_local_all is a cache-line (64B) request to the memory
-    controller, analogous to Grace's SCF cmem_rd_data.
+    Sums local + remote memory-controller requests across all chiplets. Each
+    hns_mc_reqs_*_all is a 64B cache-line request to a memory controller,
+    counted once at its home HN-S. Local counts requests served by the
+    requesting chiplet's own DMS; remote counts requests routed to the peer
+    chiplet's DMS (non-zero under NUMA-1 interleaving or cross-socket access).
     """
     mc_reqs = _sum_cmn_event(grouped_df, "hns_mc_reqs_local_all")
+    try:
+        mc_reqs = mc_reqs + _sum_cmn_event(grouped_df, "hns_mc_reqs_remote_all")
+    except KeyError:
+        # Older collect script without remote MC reqs — local-only fallback.
+        pass
+    dur = get_duration_series(grouped_df.get_group("instructions"))
+    mc_reqs.index = dur.index
+    bw_series = (mc_reqs * 64).div(dur)
+    return {
+        "name": "CMN Memory Bandwidth (MBps)",
+        "series": bw_series,
+        "prefix": 10**-6,
+    }
+
+
+@skip_if_missing
+def cmn_mem_local_bw_MBps(grouped_df):
+    """Local DRAM bandwidth — MC requests served by the requester's own chiplet DMS."""
+    mc_reqs = _sum_cmn_event(grouped_df, "hns_mc_reqs_local_all")
     dur = get_duration_series(grouped_df.get_group("instructions"))
     mc_reqs.index = dur.index
     bw_series = (mc_reqs * 64).div(dur)
     return {
-        "name": "CMN Memory Read Bandwidth (MBps)",
+        "name": "CMN Local Memory Bandwidth (MBps)",
         "series": bw_series,
         "prefix": 10**-6,
     }
 
 
+@skip_if_missing
+def cmn_mem_remote_bw_MBps(grouped_df):
+    """Remote DRAM bandwidth — MC requests routed to the peer chiplet's DMS.
+
+    Under NUMA-1 (memory interleaved across chiplets) this reflects the
+    cross-mesh share of traffic; under NUMA-2 it captures cross-socket access.
+    """
+    mc_reqs = _sum_cmn_event(grouped_df, "hns_mc_reqs_remote_all")
+    dur = get_duration_series(grouped_df.get_group("instructions"))
+    mc_reqs.index = dur.index
+    bw_series = (mc_reqs * 64).div(dur)
+    return {
+        "name": "CMN Remote Memory Bandwidth (MBps)",
+        "series": bw_series,
+        "prefix": 10**-6,
+    }
+
+
+@skip_if_missing
+def cmn_mem_read_pct(grouped_df):
+    """Approximate read share of memory traffic from HN-S PoCQ occupancy.
+
+    The HN-S has no read/write split on MC requests, but the QoS PoCQ
+    occupancy counters (read vs write) track how long read- vs write-class
+    requests sit in the point-of-coherency queue, giving a usable read/write
+    mix proxy. Reported as the read fraction of (read + write) occupancy.
+    """
+    read_occ = _sum_cmn_event(grouped_df, "hns_qos_pocq_occupancy_read")
+    write_occ = _sum_cmn_event(grouped_df, "hns_qos_pocq_occupancy_write")
+    write_occ.index = read_occ.index
+    total = read_occ + write_occ
+    return {
+        "name": "CMN Memory Read Mix %",
+        "series": read_occ.div(total),
+        "prefix": 100,
+    }
+
+
+@skip_if_missing
+def cmn_mc_retry_pct(grouped_df):
+    """Memory-controller retry rate — retried MC requests / total MC requests.
+
+    Maps to the manual's HNF "mc_retry" metric. A high retry rate indicates
+    the memory controller is backpressuring the mesh (DRAM-bound).
+    """
+    retries = _sum_cmn_event(grouped_df, "hns_mc_retries_local")
+    try:
+        retries = retries + _sum_cmn_event(grouped_df, "hns_mc_retries_remote")
+    except KeyError:
+        pass
+    reqs = _sum_cmn_event(grouped_df, "hns_mc_reqs_local_all")
+    try:
+        reqs = reqs + _sum_cmn_event(grouped_df, "hns_mc_reqs_remote_all")
+    except KeyError:
+        pass
+    retries.index = reqs.index
+    return {
+        "name": "CMN MC Retry %",
+        "series": retries.div(reqs),
+        "prefix": 100,
+    }
+
+
+@skip_if_missing
+def cmn_sf_hit_rate(grouped_df):
+    """Snoop-filter hit rate — hns_sf_hit_all / hns_slc_sf_cache_access_all.
+
+    Maps to the manual's HNF "sf_hit_ratio". Complements the existing SLC
+    (L3) hit-rate metric with coherence-directory effectiveness.
+    """
+    hit_s = _sum_cmn_event(grouped_df, "hns_sf_hit_all")
+    access_s = _sum_cmn_event(grouped_df, "hns_slc_sf_cache_access_all")
+    hit_s.index = access_s.index
+    return {
+        "name": "CMN Snoop Filter Hit Rate %",
+        "series": hit_s.div(access_s),
+        "prefix": 100,
+    }
+
+
+@skip_if_missing
+def cmn_mesh_freq_ghz(grouped_df):
+    """Mesh clock frequency (GHz) derived from the DTC cycle counter.
+
+    dtc_cycles increments at the mesh clock; dividing by the wall-clock
+    sample duration recovers the effective mesh frequency. Useful for
+    detecting mesh DVFS throttling under load.
+    """
+    cyc = _sum_cmn_event(grouped_df, "dtc_cycles")
+    # dtc_cycles is summed across chiplets; use per-chiplet average.
+    n_cmn = 0
+    for name, _ in grouped_df:
+        if isinstance(name, str) and name.endswith("/dtc_cycles/"):
+            n_cmn += 1
+    dur = get_duration_series(grouped_df.get_group("instructions"))
+    cyc.index = dur.index
+    if n_cmn > 1:
+        cyc = cyc / n_cmn
+    freq = cyc.div(dur)  # cycles per second
+    return {
+        "name": "CMN Mesh Frequency (GHz)",
+        "series": freq,
+        "prefix": 10**-9,
+    }
+
+
+# ===========================================================================
+# SVE predication effectiveness (V3-specific)
+# ===========================================================================
+
+
 @skip_if_missing
 def sve_pred_empty_pct(grouped_df):
     """SVE predicated ops with no active lanes (wasted work)."""
@@ -946,8 +1094,14 @@ def main(
         sve_pred_empty_pct(grouped_df),
         sve_pred_full_pct(grouped_df),
         sve_pred_partial_pct(grouped_df),
-        # --- CMN uncore (SLC / memory bandwidth) ---
-        cmn_mem_read_bw_MBps(grouped_df),
+        # --- CMN mesh (uncore) metrics [manual Table 15-4] ---
+        cmn_mem_bw_MBps(grouped_df),
+        cmn_mem_local_bw_MBps(grouped_df),
+        cmn_mem_remote_bw_MBps(grouped_df),
+        cmn_mem_read_pct(grouped_df),
+        cmn_mc_retry_pct(grouped_df),
+        cmn_sf_hit_rate(grouped_df),
+        cmn_mesh_freq_ghz(grouped_df),
     ]
 
     filtered_metrics = list(itertools.filterfalse(lambda x: x is None, metrics))