Skip to content

Commit 02e6d0d

Browse files
Treat explicit None filter as unrestricted for factor_library
Passing None for any factor_library filter column now removes that filter entirely and returns all values for that column (e.g. min_size_quantile=None includes all size groups), mirroring purrr::compact() on NULL in the R package. Default None values (e.g. n_portfolios_secondary) still match rows where the column is null. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 46c00ea commit 02e6d0d

4 files changed

Lines changed: 64 additions & 4 deletions

File tree

CHANGELOG.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,10 @@
77
`sorting_variable` now returns the default portfolio construction for
88
all sorting variables instead of raising an error. Other defaults
99
(e.g. `rebalancing`, `weighting_scheme`) still apply; pass
10-
`fill_all=True` to leave every column unrestricted.
10+
`fill_all=True` to leave every column unrestricted. Passing `None` for
11+
any filter column now removes that filter entirely and returns all
12+
values for that column (e.g. `min_size_quantile=None` includes all
13+
size groups), matching the R package's `NULL` behavior.
1114
- **Dependencies (replaced pyfixest with formulaic):** The `pyfixest` dependency was dropped in favor of [`formulaic`](https://github.com/matthewwardrop/formulaic) plus a small internal numpy OLS helper (`_fit_ols`). `pyfixest` was used only for plain OLS with classical (IID) standard errors in `estimate_model` and the cross-sectional / IID-variance steps of `estimate_fama_macbeth`, but it pulled in `great-tables``multimark`, a `cffi` C-extension that ships no Python 3.14 wheels and therefore required a C/C++ toolchain (e.g. MSVC Build Tools on Windows) to install on unsupported interpreters. `_fit_ols` builds the design matrix via `formulaic` and reproduces `feols` coefficients, standard errors, t-statistics, and residuals to ~1e-9 for models without fixed effects, so results are unchanged. Note that `estimate_model` and `estimate_fama_macbeth` now perform classical OLS only (the fixed-effects / clustered-SE features of `pyfixest` were never used and are no longer available).
1215

1316
## v0.1.0

tests/test_download_data_huggingface.py

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,45 @@ def test_sorting_variable_optional_returns_all_with_defaults():
284284
assert ids == [1, 2]
285285

286286

287+
def test_explicit_none_removes_filter_returning_all_values():
288+
"""Test passing None for a column returns all values for it."""
289+
grid = pd.DataFrame(
290+
{
291+
"id": [1, 2, 3],
292+
"sorting_variable": ["sv_me", "sv_me", "sv_me"],
293+
"min_size_quantile": [0.2, 0.4, 0.6],
294+
"exclude_financials": [False, False, False],
295+
"exclude_utilities": [False, False, False],
296+
"exclude_negative_earnings": [False, False, False],
297+
"sorting_variable_lag": ["6m", "6m", "6m"],
298+
"rebalancing": ["monthly", "monthly", "monthly"],
299+
"n_portfolios_main": [10, 10, 10],
300+
"sorting_method": ["univariate", "univariate", "univariate"],
301+
"n_portfolios_secondary": [None, None, None],
302+
"breakpoints_exchanges": ["NYSE", "NYSE", "NYSE"],
303+
"breakpoints_min_size_threshold": [None, None, None],
304+
"weighting_scheme": ["VW", "VW", "VW"],
305+
}
306+
)
307+
available = pd.DataFrame({"path": ["grid.parquet"], "size": [100]})
308+
with (
309+
patch(
310+
"tidyfinance.download_tidy_finance._get_available_huggingface_files",
311+
return_value=available,
312+
),
313+
patch(
314+
"tidyfinance.download_tidy_finance.pd.read_parquet",
315+
return_value=grid,
316+
),
317+
):
318+
ids = _filter_factor_library_grid(
319+
sorting_variable="me", min_size_quantile=None
320+
)
321+
322+
# The default 0.2 screen is removed, so all size groups are returned.
323+
assert ids == [1, 2, 3]
324+
325+
287326
def test_fill_all_false_defaults_applied_row_filtered_out():
288327
"""Test fill_all = FALSE: defaults applied, row filtered out."""
289328
grid = pd.DataFrame(

tidyfinance/download_tidy_finance.py

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -269,7 +269,10 @@ def _filter_factor_library_grid(fill_all: bool = False, **filters) -> list:
269269
**filters : dict
270270
Named arguments of the form 'column=value' used to filter the
271271
grid. Each value may be a scalar or a list/tuple to match multiple
272-
levels. Supported columns and their defaults are:
272+
levels. Passing 'None' for a column removes that filter entirely,
273+
returning all values for that column (e.g.,
274+
'min_size_quantile=None' includes all size groups). Supported
275+
columns and their defaults are:
273276
274277
- 'sorting_variable': no default. When omitted, all sorting
275278
variables are returned (subject to the remaining defaults).
@@ -311,8 +314,19 @@ def _filter_factor_library_grid(fill_all: bool = False, **filters) -> list:
311314
"n_portfolios_secondary must be provided."
312315
)
313316

317+
# A filter the caller explicitly sets to None is removed entirely so
318+
# all values for that column are returned, mirroring purrr::compact()
319+
# on NULL in the R implementation. These columns are recorded so the
320+
# defaults below do not reintroduce a filter for them. Default None
321+
# values (e.g. n_portfolios_secondary) are applied afterwards and
322+
# instead match rows where the column is null.
323+
unrestricted = {col for col, value in filters.items() if value is None}
324+
filters = {col: v for col, v in filters.items() if v is not None}
325+
314326
if not fill_all:
315-
filters = {**_FACTOR_LIBRARY_DEFAULTS, **filters}
327+
for col, default in _FACTOR_LIBRARY_DEFAULTS.items():
328+
if col not in filters and col not in unrestricted:
329+
filters[col] = default
316330

317331
grid = _download_factor_library_grid().assign(
318332
sorting_variable=lambda x: x["sorting_variable"].str.replace(
@@ -611,6 +625,10 @@ def _download_data_huggingface(
611625
- 'weighting_scheme' (defaults to 'VW'): return weighting within
612626
portfolios; 'VW' for value-weighted or 'EW' for equal-weighted.
613627
628+
Passing 'None' for any filter column removes that filter entirely,
629+
returning all values for that column (e.g., 'min_size_quantile=None'
630+
includes all size groups).
631+
614632
Parameters
615633
----------
616634
dataset : str

uv.lock

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)