Release 5.0.6

ODS_Tools Changelog - 5.0.6

• #271 - Fix/analysis settings empty fields
• #273 - fix relative path from_dir + cache test exposure files
• #276 - Fix MSSQL connector ignoring custom port
• #277 - enhance/data_type_optimisation
• #278 - Updated build and packaging

ODS_Tools Notes

Fix analysis settings validation for empty fields and required outputs - (PR #271)

Closes #270 — addresses two inconsistencies in analysis_settings.json validation:

• gul_summaries no longer unconditionally required: removed gul_output and gul_summaries from the schema's required array. Instead, a new check_output_summaries method validates that at least one output perspective (gul, il, ri, rl) is enabled and that each enabled output has a corresponding non-empty *_summaries list.
• output_summaries arrays enforce minItems: 1: aligns with the existing behaviour on other array fields, ensuring empty lists are consistently rejected (applies to all perspective *_summaries files)

Additional:

• Refactored setting schema testing to download schema once in the setUpClass method

fix/relative --oed-dir option - (PR #273)

• fixed relative setting for oed exposure files

Additional:

• Testing: Cache piwind exposure files in temporary directory once and use for all tests
  • local speedup tests/test_ods_package.py::OdsPackageTests 33.26s --> 11.91s (2.8x speedup)

enhance/data_type_optimisation - (PR #277)

Apply PyArrow dictionary encoding to low-cardinality string columns in the pa_dtype backend.

• oed_schema.py — bumps default OED_VERSION 4.0.0 → 5.0.0, enabling pa_dtype as the default backend.
• common.py — adds pa_dict_encode(): encodes a PyArrow-backed string series as a PyArrow dictionary using the smallest fitting index type (int8 for ≤128 unique values, int16 for ≤32767, int32 otherwise, mirroring pandas category behaviour). Uses a 2000-row head-slice pre-check to cheaply reject high-cardinality columns, then a single pc.dictionary_encode() pass that builds the dictionary and indices simultaneously (avoids the double O(n) scan of a separate nunique() + astype()). Includes a cardinality ratio guard (> 0.5) as a post-encode safety check. Uses the public __arrow_array__() protocol to access the backing PyArrow array. Failures are caught and logged at DEBUG level so the caller always keeps a usable column.
• source.py — calls pa_dict_encode() in prepare_df() for all string columns in the pa_dtype backend. Eligibility is determined at runtime by the sample pre-check rather than a static field list, so any low-cardinality varchar column is encoded automatically. Both newly-injected required columns and existing columns are covered.
• validator.py — fixes three validators that crash with TypeError on dictionary-encoded ArrowDtype columns:
  • check_conditional_requirement: extends the category dtype guard to also match dict-encoded ArrowDtype, preventing series.dtype.type(default) from being called on a dictionary type.
  • check_oedversion_consistency: decodes dict-encoded OEDVersion columns before the .str accessor call (.str does not proxy through DictionaryArray unlike pandas category).
  • check_country_and_area_code: decodes dict-encoded CountryCode/AreaCode columns before DataFrame.apply(tuple, axis=1), which does not support row iteration over DictionaryArray. The country-only path uses np.isin which handles dict-encoded columns directly and needs no decode.
• tests/test_ods_package.py — adds 9 tests: PaDictEncodeTests (6 unit tests covering low-cardinality encoding, int8/int16 index-type boundaries, high-cardinality rejection, pre-sample short-circuit, and empty series) and three regression tests for each validator fix using a pa_dtype exposure with naturally dict-encoded columns.

Known limitation

The pandas .str accessor does not work on dictionary-encoded ArrowDtype columns (unlike pandas category, which proxies .str transparently). Code that calls .str directly on OED string columns should first decode with .astype('string[pyarrow]').

---

Results

Benchmarked at N=10,000,000 rows (3 runs, median reported).

vs legacy baseline (pd_dtype + OED 4.x)

Metric	main (4.x)	branch (5.x)	Change
CSV load	42,405 ms	15,222 ms	−64%
Memory	1,011 MB	686 MB	−32%
Parquet write	15,606 ms	1,568 ms	−90%
Parquet file size	745.8 MB	298.5 MB	−60%
Parquet read	7,144 ms	231 ms	−97%

The load and I/O improvements are driven by the pa_dtype backend. The memory improvement combines pa_dtype fixing LocNumber and this PR recovering the low-cardinality regression described below.

vs pa_dtype without dictionary encoding (OED 5.x, main)

This isolates the contribution of this PR.

Metric	main (5.x, no dict)	branch (5.x)	Change
CSV load	13,509 ms	14,773 ms	+9%
Memory	1,181 MB	686 MB	−42%
Parquet write	2,094 ms	1,513 ms	−28%
Parquet file size	298.5 MB	298.5 MB	0%
Parquet read	315 ms	244 ms	−23%

Without dictionary encoding, pa_dtype uses 17% more memory than the pd_dtype baseline for typical portfolios (1,181 MB vs 1,011 MB), because low-cardinality string columns that were efficient as pandas category become plain string[pyarrow]. Dictionary encoding restores that efficiency at the cost of a +9% CSV load overhead from the runtime cardinality check and encoding pass. Parquet file size is unchanged because the parquet writer already applies RLE_DICTIONARY encoding independently of the in-memory type.

Per-column breakdown (vs pa_dtype main, 10M rows)

Column	main (MB)	branch (MB)	delta	encoding
PortNumber	124.0	9.5	−114.5	dictionary<int8, string>
AccNumber	114.4	9.5	−104.9	dictionary<int8, string>
LocPerilsCovered	104.9	9.5	−95.4	dictionary<int8, string>
LocCurrency	104.9	9.5	−95.4	dictionary<int8, string>
CountryCode	95.4	9.5	−85.9	dictionary<int8, string>
LocNumber	171.7	171.7	0	string (correctly rejected)

int8 is selected here because the benchmark data has ≤128 unique values per column. A portfolio with more unique accounts steps up to int16 (2 bytes/row) or int32 (4 bytes/row) automatically.

closes #190

Updated build and pacakging of ods-tools - (PR #278)

• Fixed build issue in CI by switching from sub-classing build_py instead of install
• Removed the --local-oed-spec flag, logic for download custom spec is moved into build workflow
• Moving to using pyproject.toml over setup.py

Release 5.0.5

ODS_Tools Changelog - 5.0.5

• #266 - Vectorize check_conditional_requirement (~127x speedup)
• #268 - use prefixItems inplace of items

ODS_Tools Notes

Fix crash in schema validation - (PR #268)

• use prefixItems inplace of items

Release 5.0.4

ODS_Tools Changelog - 5.0.4

• #238 - added oed test file generator
• #257 - enhancement/perf quantile loss sampling
• #262 - Deprecation from logger.warn -> logger.warning
• #265 - Add 'rl' perspective to model_settings_schema enums (#1495)

ODS_Tools Notes

Add OED test data generator CLI command

Add a new ods_tools generate CLI command that produces synthetic OED test data files
Loc, Acc, ReinsInfo, ReinsScope) from a JSON configuration. The generator creates files
with referential integrity across all four OED file types and supports configurable
portfolio structure, field selection, financial terms, and output format (csv/parquet).

Changes:

• ods_tools/oed/oed_generator.py — New module containing OEDFileGenerator and
  DataGenerator classes that produce synthetic OED data driven by a JSON config. Schema
  loading updated to use the existing OedSchema class.
• ods_tools/main.py — Registered the generate subcommand with options: --config,
  --output-dir, --format, --example-config, --list-versions, --list-fields, --oed-version.
• tests/data/oed_generator_config.json — Example configuration file for the generator.
• README.md — Documented all CLI commands (convert, check, transform, combine, generate).
  Previously only convert and transform were documented.

Usage:
ods_tools generate --config config.json --output-dir ./output
ods_tools generate --example-config > my_config.json
ods_tools generate --list-fields Loc --oed-version 4.0.0

perf: vectorised quantile loss sampling - (PR #257)

Replaces the pandas quantile_loss_sampling__summary_id with a vectorised NumPy implementation.

Benchmarks (1k SummaryIds, 10k Group Periods)

• combine.combine: 22.2s → 14.0s (1.6×)
• combine.sampling.quantile_loss_sampling: 18.6s → 8.58s (2.2×)

QELT assumptions
The implementation requires that QELT files have every Quantile present for every EventId (shape asserted at runtime) and that LTQuantile is in increasing order (relied on implicitly by np.digitize and indexing logic).

Release 5.0.3

ODS_Tools Changelog - 5.0.3

• #240 - enhancement/group_fill_perspective option for ods_tools combine
• #246 - enhacement/download OEDSpec during editable setup
• #247 - fix chardetv6 - dummy special chars
• #249 - Added missing token ENV to workflow script
• #251 - set default dtype to pyarrow when pandas>=3
• #252 - enhancement/combine tool performance
• #255 - fix/schema-validation
• #256 - fix/combine summaryid eventid
• #260 - auto release pr

ODS_Tools Notes

add group_fill_perspective option and add multiple analysis settings locations - (PR #240)

• Allow for group_fill_perspective=True option for combine which will fill missing il outputsets from the gul and ri outputsets from the il outpusets for the same analysis.
• Search for analysis settings in the following locations (in order of search):
  • analysis_dir/output/analysis_settings.json
  • analysis_dir/analysis_settings.json
  • analysis_dir/input/analysis_settings.json

enhacement/download OEDSpec during editable setup - (PR #246)

• pip install -e . should install missing OED Spec files

fix chardetv6 - dummy special chars - (PR #247)

• tests on non_utf8_loc.csv failing due to lack of special chars incresed the number of special chars in file

Added missing token ENV to workflow script - (PR #249)

fix #248

set default dtype to pyarrow when pandas>=3 - (PR #251)

enhancement/combine tool performance - (PR #252)

Performance optimisations and output format improvements to the ods_tools combine tool. Main improvements include:

• Faster CSV output via PyArrow (~2-3x speedup)
• Parquet output support (~3-10x speedup on file writes)
• Skip loss sampling when no overlapping events (~2.3x speedup)
• Chunked EPT generation to reduce peak memory usage (1k Summary IDs 10k Group Periods = 18GB peak memory)
• Consistent categorical dtypes across GPLT and GPQT (categorical dtypes to improve memory usage)

fix/schema-validation - (PR #255)

Fixes three bugs in schema loading and validation that caused errors when loading OED specs with certain field configurations.

Changes

ods_tools/oed/oed_schema.py

• get_default_value: treat empty string Default the same as 'n/a' — an empty Default was passed to dtype_to_python[pd_dtype]() causing an AttributeError when resolving cr_field on OED 3.0.0–3.0.3 specs
• DEV schema fallback: catch only FileNotFoundError instead of all exceptions, so unexpected errors surface rather than silently falling back to the default schema
• "latest version" lookup: use cls.DEFAULT_ODS_SCHEMA_PATH.format('*') for the glob, fixing incorrect path resolution when ODS_SCHEMA_PATH env var is set

ods_tools/oed/validator.py

• valid_values check: add .astype(bool) after .apply(is_valid_value) — pandas 3.0 no longer implicitly coerces a Categorical boolean series, causing the ~ inversion to raise a TypeError

tests/test_ods_package.py

• Fix malformed pa_dtype='int64[pyarrow' (missing ]) in test_load_oed__additional_fields
• Update dtype assertions to use exposure.backend_dtype so the test is correct under both pd_dtype and pa_dtype backends

Related

Required for OasisLMF/ODS_OpenExposureData#272
Closes #236 — AttributeError: 'float' object has no attribute 'split' on OED 3.0.0–3.0.3 specs
Closes #237 — missing JSON specs for versions < 3.0.0 (specs will be added to releases separately via ODS_OpenExposureData)

fix/combine summaryid eventid - (PR #256)

• filter group period quantile table based on summary_id, event_id present in ELT files
• keep track of already calculated summary_id, event_id to skip

CI workflow to automate Release branch - (PR #260)

Work flow that creates a release branch and PR from a given current branch, release version, and stable branch

Release 5.0.2

ODS_Tools Changelog - 5.0.2

• #158 - Add date validity checks
• #242 - ods_tools CI issues

ODS_Tools Notes

feature/date_validity_checks - (PR #218)

Adds check to ensure date columns are of format YYYY-MM-DD

closes #158

fix/CI issues - (PR #243)

• updates CI workflow file to look for renamed oasislmf build name
• fix version of chardet<=5.2.0

Release 5.0.1

• #233 - ods_tools package 5.0.0 does not include pyarrow despite needing it

ODS_Tools Notes

add pyarrow to package requirments - (PR #234)

Fix #233

Release 5.0.0

ODS_Tools Changelog - 5.0.0

• #224 - Extra OED Version sources
• #217 - error in check if missing loc acc
• #219 - Update settings schema for ktools removal
• #221 - combine ORD files
• #222 - bug/pandas v3.0 causing tests to fail

ODS_Tools Notes

feature/extra_oedversion_sources - (PR #225)

Adds option to set oed version for schema from analysis settings and from exposure files.

Order of priority

CLI  # First
Oasis config file (oasislmf.json)
analysis_settings.json
exposure files (oedversion column)
ODS_Tools version  # Last

NOTE for users: Using OED Specifications below 3.1.0 will likely cause an error due to different schema formats.

closes #224

Update settings schema for ktools removal - (PR #219)

• Removed duplicated settings validation classes
• Mark ktools only settings as depreciated
• add warning messages when validating analysis settings. (count as error if raise_warnings=True)

feature/combine ORD - (PR #220)

• Add the ods_tools combine tool which groups and combines ORD files.

bug/pandas 3 tests failing - (PR #223)

Fixes tests for pandas=3.0.0
Fixes CI errors
Fixes incorrect handling of null values from str_replace

Release 4.0.6

• #222 - bug/pandas v3.0 causing tests to fail

ODS_Tools Notes

bug/pandas 3 tests failing - (PR #223)

Fixes tests for pandas=3.0.0
Fixes CI errors
Fixes incorrect handling of null values from str_replace

Release 4.0.5

ODS_Tools Changelog - 4.0.5

• #209 - acceptance conditions for oed_schema
• #210 - support pyarrow backend dtype
• #212 - feature/allow additional fields dtypes in oed exposure
• #214 - Ci fix custom oed schema

ODS_Tools Notes

support pyarrow backend dtype - (PR #210)

Allow user to select pyarrow dtype as a backend when loading an exposure.

feature/allow additional fields dtypes in oed exposure - (PR #213)

• adds additional_fields parameter to OedExposure which can load non OED fields with the provided dtype

CI fix - load in Custom OED spec - (PR #214)

• Fixed issue where CI failed when testing package from OED-exposure branch
• Fixed Rate limit when building ods-tools from github actions

Release 4.0.4

ODS_Tools Changelog - 4.0.4

• #197 - RE: Config option for "engine" not propagating to Oasis Data Manager for pd read/write causing critical performance issues
• #200 - Added valid_pla_loss_factors_ids to event set options #199
• #201 - Fix CI build script for ODM branch select
• #202 - OEDSchema choice
• #8 - Parquet OED ReinsScope incorrectly fail validation
• #205 - Fix CI testing by pinning pip
• #206 - make nb_peril_groups_dict more friendly with numba caching function
• #207 - remove unused param

ODS_Tools Notes

Fix CI build script for ODM branch select - (PR #201)

Add input for ODM github ref when testing ODS-tools

Join Summary Info - (PR #204)

Adds option to analysis settings to allow users to set the join_summary_info field.

speed up peril filtering by making nb_peril_groups_dict more friendly with numba caching function - (PR #206)

peril filtering is used for example in fm file generation. By using this implementation, the jitted function is better cached and will be faster once it has been run one.