Unused Indexes: PgHero-style context + size sort

claude · claude · commit 7a7e09aba2ec · 2026-05-18T03:59:11.000Z
A user reported ~231 indexes flagged as unused on a real PG database, basically every non-PK/non-unique index. Investigation: idx_scan is cumulative since the stats source was last reset, so a recent PG restart or pg_stat_reset() makes everything read as "0 scans" until real traffic accumulates. We were emitting a bare JSON array of matches, with no context for the user to tell "truly unused" apart from "stats are fresh." Matches PgHero's approach: - The route now returns { indexes, stats_reset_at, min_scans } so the dashboard can render "Stats last reset 2h ago · threshold: scans ≤ 0" above the table. `stats_reset_at` comes from pg_stat_database.stats_reset on PG and is nil on MySQL (whose table_io_waits counters don't surface a reset time cheaply). - New config.min_unused_index_scans (default 0 to match PgHero). Both builders thread it through to idx_scan <= N / COUNT_READ <= N so users can ignore indexes that are technically used but rarely enough to be drop candidates. - PG results now sort by pg_relation_size(indexrelid) DESC and carry a size_bytes field; the table shows a Size column with the biggest candidates at the top (where most wins live). MySQL emits a nil size_bytes — individual index size isn't cheap to surface from information_schema.STATISTICS. - Dropped the `c.reltuples > 0` filter on PG. PgHero doesn't have it and an empty table with unused indexes is still actionable noise. The view + count widget were updated to read data.indexes; existing JSON-array consumers will need the small shape change but that's called out in CHANGELOG. IndexPlanner (which consumed UnusedIndexes directly) now pulls .indexes off the Result struct. https://claude.ai/code/session_01Xy96nK3Ron2NqukBiSgtzE
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -21,6 +21,12 @@
 - The `mysql_genius-desktop` gem stub. Desktop sidecar / macOS DMG packaging scripts (`packaging/macos/`) are removed; they only assembled the now-deleted gem.
 - `MysqlGenius::Core::VERSION` constant. Use `MysqlGenius::VERSION`.
 
+### Unused Indexes: PgHero-style context
+- `GET /mysql_genius/unused_indexes` now returns `{ indexes: [...], stats_reset_at, min_scans }` instead of a bare array. The Unused Indexes tab shows "Stats last reset N ago · threshold: scans ≤ M" above the table so an empty/fresh stats source doesn't read as "everything is unused."
+- New `config.min_unused_index_scans` (default 0 — PgHero parity). Raise it (e.g. 50) to ignore indexes that are technically used but rarely enough to consider dropping.
+- The PG query now sorts results by index byte size DESC (largest first — biggest wins). Each result hash carries a `size_bytes` field; the table shows a Size column. `pg_relation_size` is the source.
+- Removed the `c.reltuples > 0` filter so indexes on empty tables show up too (matches PgHero — empty tables with indexes are still actionable noise).
+
 ### Notes
 - PostgreSQL query stats require the `pg_stat_statements` extension to be installed and enabled (`shared_preload_libraries`).
 - Slow query log capture remains MySQL-only.
diff --git a/app/controllers/concerns/mysql_genius/database_analysis.rb b/app/controllers/concerns/mysql_genius/database_analysis.rb
@@ -26,8 +26,15 @@ def query_stats
     end
 
     def unused_indexes
-      indexes = MysqlGenius::Core::Analysis::UnusedIndexes.new(rails_connection).call
-      render(json: indexes)
+      result = MysqlGenius::Core::Analysis::UnusedIndexes.new(
+        rails_connection,
+        min_scans: mysql_genius_config.min_unused_index_scans,
+      ).call
+      render(json: {
+        indexes: result.indexes,
+        stats_reset_at: result.stats_reset_at,
+        min_scans: result.min_scans,
+      })
     rescue ActiveRecord::StatementInvalid => e
       render(json: { error: "#{unused_indexes_source_name} #{e.message.split(":").last.strip}" }, status: :unprocessable_entity)
     end
diff --git a/lib/mysql_genius/configuration.rb b/lib/mysql_genius/configuration.rb
@@ -72,6 +72,14 @@ class Configuration
     # Defaults to true.
     attr_accessor :stats_collection
 
+    # Maximum scan count for an index to still be considered "unused" by the
+    # Unused Indexes dashboard. The default (0) means only indexes that have
+    # never been scanned since the stats source was last reset are flagged.
+    # Raise this to ignore indexes that are technically used but rarely
+    # enough to be worth dropping (e.g. min_unused_index_scans = 50 to require
+    # at least 50 scans before considering an index "useful").
+    attr_accessor :min_unused_index_scans
+
     def initialize
       @featured_tables = []
       @blocked_tables = [
@@ -96,6 +104,7 @@ def initialize
       @audit_logger = nil
       @base_controller = "ActionController::Base"
       @stats_collection = true
+      @min_unused_index_scans = 0
     end
 
     def ai_enabled?
diff --git a/lib/mysql_genius/core/ai/index_planner.rb b/lib/mysql_genius/core/ai/index_planner.rb
@@ -14,7 +14,7 @@ def call(tables = nil)
           target_tables = resolve_tables(tables)
           return { "plan" => "No tables found to analyze." } if target_tables.empty?
 
-          unused = Analysis::UnusedIndexes.new(@connection).call
+          unused = Analysis::UnusedIndexes.new(@connection).call.indexes
           duplicates = Analysis::DuplicateIndexes.new(@connection, blocked_tables: []).call
           schema = SchemaContextBuilder.new(@connection).call(target_tables, detail: :with_cardinality)
 
diff --git a/lib/mysql_genius/core/analysis/unused_indexes.rb b/lib/mysql_genius/core/analysis/unused_indexes.rb
@@ -3,34 +3,58 @@
 module MysqlGenius
   module Core
     module Analysis
-      # Finds indexes with zero reads but a non-zero parent table row count.
-      # On MySQL this reads performance_schema.table_io_waits_summary_by_index_usage;
-      # on PostgreSQL it reads pg_stat_user_indexes. Returns hashes with a
-      # ready-to-run DROP INDEX statement appropriate for the dialect.
+      # Indexes whose scan count is at or below `min_scans` (default 0 — never
+      # scanned since the underlying stats source was last reset). On MySQL this
+      # reads performance_schema.table_io_waits_summary_by_index_usage; on
+      # PostgreSQL it reads pg_stat_user_indexes plus pg_relation_size for the
+      # index byte size.
       #
-      # Skips primary key indexes on both dialects. Raises if the underlying
-      # stats source is unavailable.
+      # Returns a Result with:
+      #   indexes        — Array of per-index hashes (sorted by size DESC on PG,
+      #                    by write count DESC on MySQL); each carries a dialect-
+      #                    appropriate `drop_sql` and a `size_bytes` value (nil
+      #                    on MySQL where individual index sizes aren't cheap).
+      #   stats_reset_at — Time the underlying stats source was last reset
+      #                    (PG only — pg_stat_database.stats_reset; nil on MySQL).
+      #   min_scans      — The scan threshold used for this call, echoed back so
+      #                    callers can display "indexes with ≤ N scans".
+      #
+      # Skips primary key indexes on both dialects, plus unique indexes (which
+      # are usually backing a constraint the application depends on). Raises if
+      # the underlying stats source is unavailable.
       class UnusedIndexes
-        def initialize(connection)
+        Result = Struct.new(:indexes, :stats_reset_at, :min_scans, keyword_init: true)
+
+        def initialize(connection, min_scans: 0)
           @connection = connection
           @builder = QueryBuilders.for(connection)
+          @min_scans = [min_scans.to_i, 0].max
         end
 
         def call
-          result = @connection.exec_query(@builder.unused_indexes(@connection))
+          rows = @connection.exec_query(@builder.unused_indexes(@connection, min_scans: @min_scans)).to_hashes
+          Result.new(
+            indexes: rows.map { |row| transform(row) },
+            stats_reset_at: @builder.stats_reset_at(@connection),
+            min_scans: @min_scans,
+          )
+        end
+
+        private
 
-          result.to_hashes.map do |row|
-            table = row["table_name"] || row["TABLE_NAME"]
-            index_name = row["index_name"] || row["INDEX_NAME"]
-            {
-              table: table,
-              index_name: index_name,
-              reads: (row["reads"] || row["READS"] || 0).to_i,
-              writes: (row["writes"] || row["WRITES"] || 0).to_i,
-              table_rows: (row["table_rows"] || row["TABLE_ROWS"] || 0).to_i,
-              drop_sql: @builder.drop_index_sql(table: table, index_name: index_name),
-            }
-          end
+        def transform(row)
+          table = row["table_name"] || row["TABLE_NAME"]
+          index_name = row["index_name"] || row["INDEX_NAME"]
+          size_bytes = row["size_bytes"] || row["SIZE_BYTES"]
+          {
+            table: table,
+            index_name: index_name,
+            reads: (row["reads"] || row["READS"] || 0).to_i,
+            writes: (row["writes"] || row["WRITES"] || 0).to_i,
+            table_rows: (row["table_rows"] || row["TABLE_ROWS"] || 0).to_i,
+            size_bytes: size_bytes&.to_i,
+            drop_sql: @builder.drop_index_sql(table: table, index_name: index_name),
+          }
         end
       end
     end
diff --git a/lib/mysql_genius/core/query_builders/mysql.rb b/lib/mysql_genius/core/query_builders/mysql.rb
@@ -89,27 +89,35 @@ def stats_snapshot(connection, limit:)
           SQL
         end
 
-        def unused_indexes(connection)
+        def unused_indexes(connection, min_scans: 0)
+          threshold = [min_scans.to_i, 0].max
           <<~SQL
             SELECT
               s.OBJECT_SCHEMA AS table_schema,
               s.OBJECT_NAME AS table_name,
               s.INDEX_NAME AS index_name,
               s.COUNT_READ AS `reads`,
               s.COUNT_WRITE AS `writes`,
-              t.TABLE_ROWS AS table_rows
+              t.TABLE_ROWS AS table_rows,
+              NULL AS size_bytes
             FROM performance_schema.table_io_waits_summary_by_index_usage s
             JOIN information_schema.tables t
               ON t.TABLE_SCHEMA = s.OBJECT_SCHEMA AND t.TABLE_NAME = s.OBJECT_NAME
             WHERE s.OBJECT_SCHEMA = #{connection.quote(connection.current_database)}
               AND s.INDEX_NAME IS NOT NULL
               AND s.INDEX_NAME != 'PRIMARY'
-              AND s.COUNT_READ = 0
-              AND t.TABLE_ROWS > 0
+              AND s.COUNT_READ <= #{threshold}
             ORDER BY s.COUNT_WRITE DESC
           SQL
         end
 
+        # MySQL's table_io_waits counters track since server start with no
+        # cheap way to surface that timestamp at query time, so we return nil
+        # and let the dashboard fall back to "since server restart" wording.
+        def stats_reset_at(_connection)
+          nil
+        end
+
         def drop_index_sql(table:, index_name:)
           "ALTER TABLE `#{table}` DROP INDEX `#{index_name}`;"
         end
diff --git a/lib/mysql_genius/core/query_builders/postgresql.rb b/lib/mysql_genius/core/query_builders/postgresql.rb
@@ -105,26 +105,40 @@ def stats_snapshot(connection, limit:)
           SQL
         end
 
-        def unused_indexes(_connection)
+        def unused_indexes(_connection, min_scans: 0)
+          threshold = [min_scans.to_i, 0].max
           <<~SQL
             SELECT
               s.schemaname AS table_schema,
               s.relname AS table_name,
               s.indexrelname AS index_name,
               s.idx_scan AS reads,
               s.idx_tup_read AS writes,
-              c.reltuples::bigint AS table_rows
+              c.reltuples::bigint AS table_rows,
+              pg_relation_size(s.indexrelid)::bigint AS size_bytes
             FROM pg_stat_user_indexes s
             JOIN pg_index i ON i.indexrelid = s.indexrelid
             JOIN pg_class c ON c.oid = s.relid
             WHERE NOT i.indisprimary
               AND NOT i.indisunique
-              AND s.idx_scan = 0
-              AND c.reltuples > 0
-            ORDER BY s.idx_tup_read DESC, s.indexrelname ASC
+              AND s.idx_scan <= #{threshold}
+            ORDER BY pg_relation_size(s.indexrelid) DESC, s.indexrelname ASC
           SQL
         end
 
+        # Last time pg_stat_database counters (which back pg_stat_user_indexes,
+        # pg_stat_user_tables, etc.) were reset for the current database.
+        # Surfacing this lets the dashboard distinguish "this index is unused"
+        # from "stats were reset five minutes ago and nothing has run yet".
+        def stats_reset_at(connection)
+          connection.select_value(
+            "SELECT stats_reset FROM pg_stat_database " \
+              "WHERE datname = #{connection.quote(connection.current_database)}",
+          )
+        rescue StandardError
+          nil
+        end
+
         def drop_index_sql(table:, index_name:)
           _ = table
           %(DROP INDEX IF EXISTS "#{index_name.to_s.gsub('"', '""')}";)
diff --git a/lib/mysql_genius/core/views/mysql_genius/queries/_tab_unused_indexes.html.erb b/lib/mysql_genius/core/views/mysql_genius/queries/_tab_unused_indexes.html.erb
@@ -20,6 +20,7 @@
       </div>
     </div>
   </div>
+  <div id="unused-stats-context" class="mg-text-muted mg-hidden" style="font-size:12px;margin-bottom:8px;"></div>
   <div id="unused-loading" class="mg-text-center mg-hidden"><span class="mg-spinner"></span> Scanning...</div>
   <div id="unused-error" class="mg-hidden"></div>
   <div id="unused-empty" class="mg-text-center mg-text-muted mg-hidden">No unused indexes found.</div>
@@ -29,6 +30,7 @@
         <tr>
           <th>Table</th>
           <th>Index</th>
+          <th style="text-align:right">Size</th>
           <th style="text-align:right">Reads</th>
           <th style="text-align:right">Writes</th>
           <th style="text-align:right">Table Rows</th>
diff --git a/lib/mysql_genius/core/views/mysql_genius/queries/dashboard.html.erb b/lib/mysql_genius/core/views/mysql_genius/queries/dashboard.html.erb
@@ -89,6 +89,47 @@
   function hide(e) { e.classList.add('mg-hidden'); }
   function escHtml(s) { var d = document.createElement('div'); d.textContent = s; return d.innerHTML; }
 
+  function formatBytes(bytes) {
+    if (bytes === null || bytes === undefined) return '—';
+    var n = Number(bytes);
+    if (!isFinite(n) || n <= 0) return '0 B';
+    var units = ['B', 'KB', 'MB', 'GB', 'TB'];
+    var i = Math.min(units.length - 1, Math.floor(Math.log(n) / Math.log(1024)));
+    return (n / Math.pow(1024, i)).toFixed(i === 0 ? 0 : 1) + ' ' + units[i];
+  }
+
+  function formatRelativeTime(isoString) {
+    if (!isoString) return null;
+    var then = new Date(isoString);
+    if (isNaN(then.getTime())) return null;
+    var diffMs = Date.now() - then.getTime();
+    if (diffMs < 0) diffMs = 0;
+    var seconds = Math.floor(diffMs / 1000);
+    if (seconds < 60) return seconds + 's ago';
+    var minutes = Math.floor(seconds / 60);
+    if (minutes < 60) return minutes + 'm ago';
+    var hours = Math.floor(minutes / 60);
+    if (hours < 48) return hours + 'h ago';
+    var days = Math.floor(hours / 24);
+    return days + 'd ago';
+  }
+
+  function renderUnusedStatsContext(data) {
+    var ctx = el('unused-stats-context');
+    if (!ctx) return;
+    var parts = [];
+    if (data && data.stats_reset_at) {
+      var rel = formatRelativeTime(data.stats_reset_at);
+      if (rel) parts.push('Stats last reset <strong>' + escHtml(rel) + '</strong>');
+    }
+    var threshold = data && (data.min_scans || data.min_scans === 0) ? data.min_scans : null;
+    if (threshold !== null) {
+      parts.push('threshold: scans &le; <strong>' + threshold + '</strong>');
+    }
+    ctx.innerHTML = parts.length ? parts.join(' &middot; ') : '';
+    if (parts.length) show(ctx); else hide(ctx);
+  }
+
   // --- Table Sorting ---
 
   function parseSortValue(text) {
@@ -459,7 +500,8 @@
 
     // Unused indexes count
     ajaxGet(ROUTES.unused_indexes, {}, function(data) {
-      var count = Array.isArray(data) ? data.length : 0;
+      var indexes = (data && data.indexes) || [];
+      var count = indexes.length;
       var countEl = el('dash-unused-count');
       if (count === 0) {
         countEl.innerHTML = '<span style="color:#28a745;">&#10003; 0</span>';
@@ -1142,12 +1184,15 @@
         el('unused-error').innerHTML = '<div class="mg-alert mg-alert-warning">' + escHtml(data.error) + '</div>';
         show(el('unused-error')); return;
       }
-      if (!data.length) { show(el('unused-empty')); el('unused-count').textContent = '0'; return; }
-      el('unused-count').textContent = data.length + ' found';
-      el('unused-tbody').innerHTML = data.map(function(d) {
+      var indexes = (data && data.indexes) || [];
+      renderUnusedStatsContext(data);
+      if (!indexes.length) { show(el('unused-empty')); el('unused-count').textContent = '0'; return; }
+      el('unused-count').textContent = indexes.length + ' found';
+      el('unused-tbody').innerHTML = indexes.map(function(d) {
         return '<tr>' +
           '<td><strong>' + escHtml(d.table) + '</strong></td>' +
           '<td><code>' + escHtml(d.index_name) + '</code></td>' +
+          '<td class="mg-num">' + formatBytes(d.size_bytes) + '</td>' +
           '<td class="mg-num">' + d.reads + '</td>' +
           '<td class="mg-num">' + Number(d.writes).toLocaleString() + '</td>' +
           '<td class="mg-num">' + Number(d.table_rows).toLocaleString() + '</td>' +
@@ -1161,7 +1206,7 @@
       var ts = migrationTimestamp();
       var migrationLines = ['# ' + ts + '_remove_unused_indexes.rb', '',
         'class RemoveUnusedIndexes < ActiveRecord::Migration[' + RAILS_MIGRATION_VERSION + ']', '  def change'];
-      data.forEach(function(d) {
+      indexes.forEach(function(d) {
         migrationLines.push('    remove_index :' + d.table + ', name: :' + d.index_name);
       });
       migrationLines.push('  end', 'end');
diff --git a/spec/mysql_genius/configuration_spec.rb b/spec/mysql_genius/configuration_spec.rb
@@ -28,6 +28,10 @@
       expect(config.slow_query_threshold_ms).to(eq(250))
     end
 
+    it "defaults min_unused_index_scans to 0 (PgHero parity — flag only truly unused)" do
+      expect(config.min_unused_index_scans).to(eq(0))
+    end
+
     it "defaults authenticate to allow all" do
       expect(config.authenticate.call(nil)).to(be(true))
     end
diff --git a/spec/mysql_genius/core/analysis/unused_indexes_spec.rb b/spec/mysql_genius/core/analysis/unused_indexes_spec.rb
diff --git a/spec/requests/mysql_genius/analysis_spec.rb b/spec/requests/mysql_genius/analysis_spec.rb
