summarize_stops: empty and summarized stop tables have different schemas

[paco-barreras]

This issue includes analysis written with the assistance of AI. The code has not yet been reviewed by a human (remove this disclosure after human review).

What happens

The same function returns a different table depending on whether any stop was found.

import pandas as pd, numpy as np
from nomad.stop_detection.sequential_algs import lachesis

n = 8
df = pd.DataFrame({
    "timestamp": np.arange(n) * 60 + 1700000000,
    "x": np.zeros(n), "y": np.zeros(n), "ha": np.ones(n),
})

real     = lachesis(df, delta_roam=100, dt_max=60, dur_min=1,     complete_output=True)  # one stop
no_stops = lachesis(df, delta_roam=100, dt_max=60, dur_min=10**6, complete_output=True)  # no stop qualifies

	columns	timestamp	x	max_gap
real	cluster, x, y, timestamp, ha, diameter, n_pings, end_timestamp, duration, max_gap	int64	float64	float64
no_stops	x, y, timestamp, ha, diameter, n_pings, end_timestamp, duration, max_gap	Int64	Float64	Int64

The cluster column is missing from the second, and every dtype belongs to a different family (numpy vs pandas nullable). Stacking the two gives a table that matches neither:

pd.concat([no_stops, real])
# 'cluster': 'float64'   <- new all-NaN column, because one side lacked it
# 'x': 'float64', 'timestamp': 'Int64', 'max_gap': 'float64'   <- mixed families

An empty input frame takes the same path as "no stop qualified", so this is reachable from *_per_user wrappers, and from any loop that summarises per day or per time bucket where one slice happens to contain no stop.

Why it happens

Three functions are involved:

• summarize_stop builds one row for one cluster. It returns pd.Series(stop_attr, dtype="object").
• summarize_stops is the driver: it joins the labels onto the pings, drops noise (cluster == -1), groups by cluster, calls summarize_stop on each group via groupby(..., as_index=False).apply(...), and finally drops stops shorter than dur_min.
• _get_empty_stop_df builds an empty stop table from scratch, and is called by summarize_stops when no clusters remain.

DataFrameGroupBy.apply determines the resulting columns and dtypes by inspecting what the callback returned. When there are zero groups the callback is never invoked, so pandas has nothing to inspect and falls back to the columns of the input frame:

df.groupby("cluster", as_index=False).apply(summarize, include_groups=False)
# 3 rows in  -> ['cluster', 'x', 'start_timestamp', 'duration']   (the summariser's columns)
# 0 rows in  -> ['timestamp', 'x']                                (the input's columns)

That fallback is why _get_empty_stop_df exists. The scope is narrow: groups produced by a plain-column groupby always contain at least one row, so the only degenerate case is no clusters at all.

The consequence is that the stop table's schema is specified twice:

1. implicitly by summarize_stop, as whatever pandas infers from the values it returns, plus a cluster column contributed by as_index=False;
2. explicitly by _get_empty_stop_df, as a hand-maintained mapping from column to dtype, which has no cluster column and uses nullable dtypes.

Nobody keeps the two in step, and they now disagree on the column list and on every dtype.

The summarised schema is not stable either

Because summarize_stop returns an object Series, the final dtypes depend on the values that happened to be produced:

• diameter is the integer 0 for a one-ping stop (_diameter early-returns 0) and float64 for a multi-ping stop;
• max_gap comes out float64 when diff() introduced a NaN and int otherwise.

So there is no fixed schema for _get_empty_stop_df to imitate, which is why keeping the two in sync cannot work as a strategy.

Why the existing tests did not catch it

• test_get_empty_stop_df_basic and the other 13 test_get_empty_stop_df_* tests (test_stop_detection_utils.py) call _get_empty_stop_df directly and compare against hardcoded expected dtypes. They pin the empty path against itself and never call summarize_stops, so they cannot observe a disagreement between the two paths.
• test_empty_dataframe_complete_output and test_empty_dataframe_xy_output (test_stop_detection.py) assert set(result.columns) == expected_cols, again a hardcoded expectation for the empty path only. Using a set also makes column order invisible.
• test_get_empty_stop_df_matches_summarize_stop_schema_for_empty_and_clustered_input was the only test that intended to compare the two paths, and it failed to for two reasons: it called _summarize_stop_clusters, a copy of the driver defined inside the test module, which grouped without as_index=False and therefore produced no cluster column on either side; and it compared column names only, never dtypes.
• test_stop_output_is_valid_stop_df validates with _is_stop_df, which ignores non-canonical columns and deliberately accepts both numpy and nullable dtypes, so it passes for either schema.

The replacement, test_summarize_stops_empty_output_matches_summarized_schema, calls the real summarize_stops twice on the same frame (labels producing two clusters, and all-noise labels) and asserts equality of both columns and dtypes, parametrized over complete_output x keep_col_names. It is xfail(strict=True) and records this bug.

How other libraries handle the same problem

This is a known consequence of schema inference, and there are two established answers:

• PySpark makes the schema mandatory: applyInPandas(func, schema=...) will not run without one, because a distributed engine cannot infer a schema from partitions that may be empty.
• Dask takes an optional meta argument, which is literally an empty frame carrying the intended dtypes. When it is not supplied, dask.dataframe.utils.meta_nonempty fabricates a small synthetic frame with dummy values appropriate to each dtype, runs the function on it, and adopts the resulting schema. It uses two rows, not one, because many operations degenerate on a single row.

_get_empty_stop_df is nomad's hand-written meta. The problem is not that it exists, it is that it is a second specification rather than the specification. The synthetic-frame route was also considered and is worth recording as rejected for now: a one-row probe already crashes this code (see the related bug below), and matching Dask would mean generating dummy values per dtype while also satisfying the column-resolution rules in _fallback_st_cols.

Agreed design decisions

1. The cluster label column stays named cluster. Renaming it to the canonical label key of DEFAULT_SCHEMA is deferred to a separate issue.
2. Stop-table columns adopt pandas nullable dtypes. nomad.io.base already accepts them on input and coerces timestamps to Int64 on load, so outputs should meet the same standard the library accepts as input.
3. diameter, n_pings and max_gap become canonical schema columns and are all integers: diameter in whole metres, max_gap in whole minutes.

Plan

1. Lift the key-to-dtype mapping currently hardcoded inside _is_stop_df into a single constant, and have _is_stop_df validate against it rather than restate it. Add the three statistics columns to DEFAULT_SCHEMA as integer keys.
2. Add one schema function in nomad/stop_detection/utils.py returning an ordered {column: dtype} from traj_cols, complete_output, keep_col_names, passthrough_cols and the input frame. Canonical columns take their dtype from that constant, with time columns inheriting the input's datetime dtype so the timezone survives; passthrough columns inherit data[col].dtype.
3. summarize_stops uses that schema on both paths: cast the summarised frame to it, or emit it with zero rows when there are no clusters. summarize_stop_grid uses the same function so the grid-based path cannot drift either.
4. Delete _get_empty_stop_df together with its tests, and remove the xfail.

No synthetic rows are needed: canonical dtypes are declared, and default passthrough dtypes are copied from the input frame.

Interaction with #415

#415 adds passthrough_agg, applying grouped_data[col].agg(func) per passthrough column. It reuses the passthrough column's own name, so it introduces no unknown column names, but it does introduce unknown dtypes: the same Int64 column yields float64 under mean, a Python int under nunique, and int64 under max. Those columns are the one place where a declared dtype, or a Dask-style two-row probe, is genuinely required.

Two further points to reconcile when #415 merges:

• its summarize_stops groups without as_index=False, dropping the cluster column that decision 1 keeps;
• it calls _get_empty_stop_df(data.columns, ...), whose first parameter is now the frame rather than the column index.

Related bug, fixed separately

summarize_stop raised AttributeError: 'int' object has no attribute 'total_seconds' for a one-ping stop with datetime input and complete_output=True, because max_gap fell back to the integer 0 and was then treated as a Timedelta. Fixed by falling back to pd.Timedelta(0) on the datetime path, with a regression test covering both time representations.