feat(hparams): shared filter sidebar and unified axis control across views ( Layer 3 - 6)

.agents/skills/env-persistence/SKILL.md

@@ -28,6 +28,8 @@ Use this skill for persistence format changes and env reload/view layout behavio
 ## Documentation
 
 - [Skill reference](references/REFERENCE.md)
+- `py/visdom/data_model/README.md` — the storage abstraction layer (`DataStore`/`JSONStore`); persistence now funnels through it
+- `py/visdom/data_model/base.py`, `py/visdom/data_model/json_store.py`
 - `py/visdom/utils/server_utils.py`
 - `py/visdom/server/handlers/web_handlers.py`
 - `py/visdom/server/app.py`

README.md

@@ -358,6 +358,15 @@ vis._send({'data': [trace], 'layout': layout, 'win': 'mywin'})
 - [`vis.check_connection`](#vischeck_connection): check if the server is connected
 - [`vis.replay_log`](#visreplay_log): replay the actions from the provided log file
 
+### Experiments
+Track experiment metadata (hyper-parameters, metrics, tags) alongside your plots, then search and compare runs across your server:
+- [`vis.experiment`](#visexperiment)  : create or update experiment metadata for an env
+- [`vis.log_metrics`](#vislog_metrics)  : append metric observations to an env's experiment
+- [`vis.finish_experiment`](#visfinish_experiment)  : mark an experiment terminal (finished/failed)
+- [`vis.search_experiments`](#vissearch_experiments)  : search experiments across envs with a query
+- [`vis.compare_experiments`](#viscompare_experiments)  : diff experiments field by field
+- [`vis.suggest_experiment`](#vissuggest_experiment)  : suggest parameters for the next run (reserved)
+
 
 ## Loggers
 
@@ -1106,6 +1115,75 @@ This function takes the contents of a visdom log and replays them to the current
 Arguments:
 - `log_filename`: log file to replay the contents of.
 
+### Experiments
+
+Attach experiment metadata — hyper-parameters, metric observations and tags — to an environment, then search and compare runs across your server. Metadata is stored under the environment's `experiment` key and persisted through the server's data store, so a server started without a persistence path (`env_path=None`) has nothing to store, search or compare. Each function returns the server's reply decoded as a dict.
+
+#### vis.experiment
+
+This function creates or updates the experiment metadata for an environment. Calling it again for the same env merges in new params/tags and overwrites the name/description, so it is safe to call at the start of and again during a run.
+
+Arguments:
+- `name`: Display name for the experiment. Defaults to the env id.
+- `params`: Hyper-parameters as a dict of `{name: value}`.
+- `tags`: Free-form tags as a dict of `{name: value}`.
+- `description`: Free-form description.
+- `env`: Environment to attach the experiment to. Defaults to the client's env.
+
+#### vis.log_metrics
+
+This function appends one or more metric observations to an env's experiment, creating the experiment automatically if it does not exist yet.
+
+Arguments:
+- `metrics`: A non-empty dict of `{name: value}` observations.
+- `step`: Optional training step the observations were recorded at.
+- `env`: Environment whose experiment to log to. Defaults to the client's env.
+
+> **Note**: once an experiment is finished (see `finish_experiment`), further `experiment`/`log_metrics` writes are rejected so a completed run's recorded data cannot change after the fact.
+
+#### vis.finish_experiment
+
+This function marks an env's experiment terminal so its recorded data is frozen.
+
+Arguments:
+- `status`: Terminal status, either `finished` (default) or `failed`.
+- `env`: Environment whose experiment to finish. Defaults to the client's env.
+
+#### vis.search_experiments
+
+This function searches the experiments logged on the server, across all environments. The `query` uses a small readable syntax — comparisons (`<`, `<=`, `>`, `>=`, `=`, `!=`, `contains`) over param, metric and tag names, combined with `AND`/`OR` and parentheses. A name is matched bare (`acc`) or namespaced (`metric.acc`, `param.lr`, `tag.owner`) when ambiguous; metrics compare on their latest value. Queries are evaluated in Python, never `eval`'d, so a hostile query is a parse error rather than code execution.
+
+```python
+vis.search_experiments("lr < 0.01 AND acc > 0.9")
+vis.search_experiments("status = finished AND (dataset contains mnist)")
+```
+
+Arguments:
+- `query`: The filter string. `None` (default) returns everything.
+- `limit`: Maximum results to return (default `100`). `None` returns all matches.
+- `offset`: Number of results to skip, for paging (default `0`).
+- `sort_by`: Field to sort by (any of the same names). Defaults to newest-created first.
+- `descending`: Sort direction (default `True`).
+
+Returns a dict of `experiments` (one page's worth), the unpaged `total` matching the query, and the `limit`/`offset`/`query` used.
+
+#### vis.compare_experiments
+
+This function compares the named experiments field by field to see what differs. Finding the runs is `search_experiments`' job — it answers "which runs match?"; this answers "how do these runs differ?". To compare a query's matches, search first and pass the resulting ids on.
+
+Arguments:
+- `env_ids`: A list (or tuple) of environment ids to compare, in the order given. Each must have an experiment.
+
+Returns a dict with the compared runs (`env_ids`, full `experiments`) and a `params`, `metrics` and `tags` section. Each section lists the union of `fields`, the `shared` ones every run agrees on, the `differing` rest, the per-run `values`, and `groups` clustering the runs that agree per field.
+
+#### vis.suggest_experiment
+
+> **Reserved**: this endpoint is a stub. Choosing the next set of hyper-parameters to try (an Optuna-backed search strategy) is planned for a later layer, so the server currently replies `501 Not Implemented` and this function returns a stub dict `{"status": "not_implemented", "suggestion": None, ...}` rather than a real suggestion. The method, its arguments and the endpoint are in place so callers and docs are ready for when the strategy is wired in.
+
+Arguments:
+- `params`: The search space to suggest over, as a dict of `{name: spec}`. Currently ignored by the stub.
+- `env`: Environment (study) to suggest against. Defaults to the client's env.
+
 ## Customizing Visdom
 The user config directory for visdom is
 - `~/.config/visdom` for Linux

js/main.js

@@ -12,6 +12,7 @@
 'use strict';
 
 import 'fetch';
+import 'rc-slider/assets/index.css';
 import 'rc-tree-select/assets/index.less';
 
 import React, { useContext, useEffect, useRef, useState } from 'react';

js/panes/HParamsPane.js

@@ -0,0 +1,254 @@
+/**
+ * Copyright 2017-present, The Visdom Authors
+ * All rights reserved.
+ *
+ * This source code is licensed under the license found in the
+ * LICENSE file in the root directory of this source tree.
+ *
+ */
+
+import React, { useMemo, useState } from 'react';
+
+import HParamsFilters from './hparams/HParamsFilters';
+import HParamsParallelCoords from './hparams/HParamsParallelCoords';
+import HParamsSplom from './hparams/HParamsSplom';
+import HParamsTable from './hparams/HParamsTable';
+import {
+  applyFilters,
+  buildColumns,
+  buildFilterSpecs,
+  collectStatuses,
+  countActiveFilters,
+  filterRecords,
+} from './hparams/hparamsUtils';
+import Pane from './Pane';
+
+const VIEWS = [
+  { key: 'table', label: 'Table' },
+  { key: 'parcoords', label: 'Parallel coordinates' },
+  { key: 'splom', label: 'Scatter matrix' },
+];
+
+const NO_RECORDS = [];
+
+function readContent(content) {
+  if (!content || typeof content !== 'object' || Array.isArray(content)) {
+    return null;
+  }
+  if (!Array.isArray(content.records)) {
+    return null;
+  }
+  const asKeys = (value) => (Array.isArray(value) ? value : []);
+  return {
+    records: content.records,
+    paramKeys: asKeys(content.param_keys),
+    metricKeys: asKeys(content.metric_keys),
+    tagKeys: asKeys(content.tag_keys),
+  };
+}
+
+var HParamsPane = (props) => {
+  const { content } = props;
+  const data = readContent(content);
+  const [view, setView] = useState('table');
+  const [tableSort, setTableSort] = useState({ by: null, dir: null });
+  const [tableFilter, setTableFilter] = useState('');
+  const [tableColorBy, setTableColorBy] = useState(null);
+  const [tableSelected, setTableSelected] = useState(() => new Set());
+  const [splomDims, setSplomDims] = useState(null);
+  const [splomColorBy, setSplomColorBy] = useState(null);
+  const [parcoordsDims, setParcoordsDims] = useState(null);
+  const [parcoordsColorBy, setParcoordsColorBy] = useState(null);
+  const [filtersOpen, setFiltersOpen] = useState(true);
+  const [filters, setFilters] = useState({ statuses: [], columns: {} });
+
+  const records = data ? data.records : NO_RECORDS;
+  const columns = useMemo(
+    () =>
+      data ? buildColumns(data.paramKeys, data.metricKeys, data.tagKeys) : [],
+    [data]
+  );
+  const specs = useMemo(
+    () => buildFilterSpecs(records, columns),
+    [records, columns]
+  );
+  const statuses = useMemo(() => collectStatuses(records), [records]);
+  const visibleRecords = useMemo(
+    () =>
+      applyFilters(
+        filterRecords(records, tableFilter, columns),
+        specs,
+        filters
+      ),
+    [records, tableFilter, columns, specs, filters]
+  );
+  const activeFilters =
+    countActiveFilters(filters, specs) + (tableFilter.trim() ? 1 : 0);
+
+  const handleDownload = () => {
+    let blob = new Blob([JSON.stringify(content)], {
+      type: 'application/json',
+    });
+    let url = window.URL.createObjectURL(blob);
+    let link = document.createElement('a');
+    link.download = 'visdom_hparams.json';
+    link.href = url;
+    link.click();
+  };
+
+  let body;
+  if (data === null) {
+    body = (
+      <div className="hparams-message hparams-error">
+        Could not read hyper-parameter data for this window.
+      </div>
+    );
+  } else if (data.records.length === 0) {
+    body = (
+      <div className="hparams-message hparams-empty">
+        No experiments match this selection.
+      </div>
+    );
+  } else {
+    body = (
+      <div className="hparams-body">
+        <div className="hparams-summary">
+          <span className="hparams-stat">
+            <b>{data.records.length}</b> runs
+          </span>
+          <span className="hparams-stat">
+            <b>{data.paramKeys.length}</b> params
+          </span>
+          <span className="hparams-stat">
+            <b>{data.metricKeys.length}</b> metrics
+          </span>
+          <span className="hparams-stat">
+            <b>{data.tagKeys.length}</b> tags
+          </span>
+          {visibleRecords.length !== records.length ? (
+            <span className="hparams-stat hparams-stat-filtered">
+              showing <b>{visibleRecords.length}</b> of {records.length}
+            </span>
+          ) : null}
+        </div>
+        <div className="hparams-views">
+          <div className="hparams-viewtabs" role="tablist">
+            {VIEWS.map((v) => (
+              <button
+                key={v.key}
+                type="button"
+                role="tab"
+                aria-selected={view === v.key}
+                className={
+                  'hparams-viewtab' +
+                  (view === v.key ? ' hparams-viewtab-active' : '')
+                }
+                onClick={() => setView(v.key)}
+              >
+                {v.label}
+              </button>
+            ))}
+            <span className="hparams-viewtools">
+              <input
+                type="text"
+                className="hparams-filter"
+                placeholder="Search runs…"
+                value={tableFilter}
+                onChange={(e) => setTableFilter(e.target.value)}
+                aria-label="Search runs"
+              />
+              <button
+                type="button"
+                className={
+                  'hparams-filters-toggle' +
+                  (filtersOpen ? ' hparams-filters-toggle-active' : '')
+                }
+                onClick={() => setFiltersOpen((open) => !open)}
+                aria-expanded={filtersOpen}
+                title="Filter runs across every view"
+              >
+                Filters{activeFilters ? ' (' + activeFilters + ')' : ''}
+              </button>
+            </span>
+          </div>
+          <div className="hparams-layout">
+            {filtersOpen ? (
+              <HParamsFilters
+                specs={specs}
+                statuses={statuses}
+                filters={filters}
+                setFilters={setFilters}
+                setSearch={setTableFilter}
+                onClose={() => setFiltersOpen(false)}
+                visibleCount={visibleRecords.length}
+                totalCount={records.length}
+              />
+            ) : null}
+            {(() => {
+              if (visibleRecords.length === 0) {
+                return (
+                  <div className="hparams-message hparams-empty">
+                    No runs match your filters.
+                  </div>
+                );
+              }
+              const viewProps = {
+                records: visibleRecords,
+                columnRecords: records,
+                paramKeys: data.paramKeys,
+                metricKeys: data.metricKeys,
+                tagKeys: data.tagKeys,
+              };
+              if (view === 'splom')
+                return (
+                  <HParamsSplom
+                    {...viewProps}
+                    selectedDims={splomDims}
+                    onSelectedDims={setSplomDims}
+                    colorBy={splomColorBy}
+                    onColorBy={setSplomColorBy}
+                  />
+                );
+              if (view === 'parcoords')
+                return (
+                  <HParamsParallelCoords
+                    {...viewProps}
+                    selectedDims={parcoordsDims}
+                    onSelectedDims={setParcoordsDims}
+                    colorBy={parcoordsColorBy}
+                    onColorBy={setParcoordsColorBy}
+                  />
+                );
+              return (
+                <HParamsTable
+                  {...viewProps}
+                  sort={tableSort}
+                  setSort={setTableSort}
+                  colorBy={tableColorBy}
+                  setColorBy={setTableColorBy}
+                  selected={tableSelected}
+                  setSelected={setTableSelected}
+                />
+              );
+            })()}
+          </div>
+        </div>
+      </div>
+    );
+  }
+
+  return (
+    <Pane {...props} handleDownload={handleDownload}>
+      <div className="content-hparams">{body}</div>
+    </Pane>
+  );
+};
+
+HParamsPane = React.memo(HParamsPane, (props, nextProps) => {
+  if (props.contentID !== nextProps.contentID) return false;
+  else if (props.h !== nextProps.h || props.w !== nextProps.w) return false;
+  else if (props.isFocused !== nextProps.isFocused) return false;
+  return true;
+});
+
+export default HParamsPane;

js/panes/Pane.js

@@ -82,8 +82,14 @@ var Pane = forwardRef((props, ref) => {
 
   // rendering
   // ---------
-  let windowClassNames = classNames({ window: true, focus: props.isFocused });
-  let barClassNames = classNames({ bar: true, focus: props.isFocused });
+  let windowClassNames = classNames(
+    { window: true, focus: props.isFocused },
+    props.type && 'window-' + props.type
+  );
+  let barClassNames = classNames(
+    { bar: true, focus: props.isFocused },
+    props.type && 'bar-' + props.type
+  );
 
   let contentClassNames = classNames({
    content: true,