Release 5.0.6

ODS_Tools Changelog - 5.0.6

• #271 - Fix/analysis settings empty fields
• #273 - fix relative path from_dir + cache test exposure files
• #276 - Fix MSSQL connector ignoring custom port
• #277 - enhance/data_type_optimisation
• #278 - Updated build and packaging

ODS_Tools Notes

Fix analysis settings validation for empty fields and required outputs - (PR #271)

Closes #270 — addresses two inconsistencies in analysis_settings.json validation:

• gul_summaries no longer unconditionally required: removed gul_output and gul_summaries from the schema's required array. Instead, a new check_output_summaries method validates that at least one output perspective (gul, il, ri, rl) is enabled and that each enabled output has a corresponding non-empty *_summaries list.
• output_summaries arrays enforce minItems: 1: aligns with the existing behaviour on other array fields, ensuring empty lists are consistently rejected (applies to all perspective *_summaries files)

Additional:

• Refactored setting schema testing to download schema once in the setUpClass method

fix/relative --oed-dir option - (PR #273)

• fixed relative setting for oed exposure files

Additional:

• Testing: Cache piwind exposure files in temporary directory once and use for all tests
  • local speedup tests/test_ods_package.py::OdsPackageTests 33.26s --> 11.91s (2.8x speedup)

enhance/data_type_optimisation - (PR #277)

Apply PyArrow dictionary encoding to low-cardinality string columns in the pa_dtype backend.

• oed_schema.py — bumps default OED_VERSION 4.0.0 → 5.0.0, enabling pa_dtype as the default backend.
• common.py — adds pa_dict_encode(): encodes a PyArrow-backed string series as a PyArrow dictionary using the smallest fitting index type (int8 for ≤128 unique values, int16 for ≤32767, int32 otherwise, mirroring pandas category behaviour). Uses a 2000-row head-slice pre-check to cheaply reject high-cardinality columns, then a single pc.dictionary_encode() pass that builds the dictionary and indices simultaneously (avoids the double O(n) scan of a separate nunique() + astype()). Includes a cardinality ratio guard (> 0.5) as a post-encode safety check. Uses the public __arrow_array__() protocol to access the backing PyArrow array. Failures are caught and logged at DEBUG level so the caller always keeps a usable column.
• source.py — calls pa_dict_encode() in prepare_df() for all string columns in the pa_dtype backend. Eligibility is determined at runtime by the sample pre-check rather than a static field list, so any low-cardinality varchar column is encoded automatically. Both newly-injected required columns and existing columns are covered.
• validator.py — fixes three validators that crash with TypeError on dictionary-encoded ArrowDtype columns:
  • check_conditional_requirement: extends the category dtype guard to also match dict-encoded ArrowDtype, preventing series.dtype.type(default) from being called on a dictionary type.
  • check_oedversion_consistency: decodes dict-encoded OEDVersion columns before the .str accessor call (.str does not proxy through DictionaryArray unlike pandas category).
  • check_country_and_area_code: decodes dict-encoded CountryCode/AreaCode columns before DataFrame.apply(tuple, axis=1), which does not support row iteration over DictionaryArray. The country-only path uses np.isin which handles dict-encoded columns directly and needs no decode.
• tests/test_ods_package.py — adds 9 tests: PaDictEncodeTests (6 unit tests covering low-cardinality encoding, int8/int16 index-type boundaries, high-cardinality rejection, pre-sample short-circuit, and empty series) and three regression tests for each validator fix using a pa_dtype exposure with naturally dict-encoded columns.

Known limitation

The pandas .str accessor does not work on dictionary-encoded ArrowDtype columns (unlike pandas category, which proxies .str transparently). Code that calls .str directly on OED string columns should first decode with .astype('string[pyarrow]').

---

Results

Benchmarked at N=10,000,000 rows (3 runs, median reported).

vs legacy baseline (pd_dtype + OED 4.x)

Metric	main (4.x)	branch (5.x)	Change
CSV load	42,405 ms	15,222 ms	−64%
Memory	1,011 MB	686 MB	−32%
Parquet write	15,606 ms	1,568 ms	−90%
Parquet file size	745.8 MB	298.5 MB	−60%
Parquet read	7,144 ms	231 ms	−97%

The load and I/O improvements are driven by the pa_dtype backend. The memory improvement combines pa_dtype fixing LocNumber and this PR recovering the low-cardinality regression described below.

vs pa_dtype without dictionary encoding (OED 5.x, main)

This isolates the contribution of this PR.

Metric	main (5.x, no dict)	branch (5.x)	Change
CSV load	13,509 ms	14,773 ms	+9%
Memory	1,181 MB	686 MB	−42%
Parquet write	2,094 ms	1,513 ms	−28%
Parquet file size	298.5 MB	298.5 MB	0%
Parquet read	315 ms	244 ms	−23%

Without dictionary encoding, pa_dtype uses 17% more memory than the pd_dtype baseline for typical portfolios (1,181 MB vs 1,011 MB), because low-cardinality string columns that were efficient as pandas category become plain string[pyarrow]. Dictionary encoding restores that efficiency at the cost of a +9% CSV load overhead from the runtime cardinality check and encoding pass. Parquet file size is unchanged because the parquet writer already applies RLE_DICTIONARY encoding independently of the in-memory type.

Per-column breakdown (vs pa_dtype main, 10M rows)

Column	main (MB)	branch (MB)	delta	encoding
PortNumber	124.0	9.5	−114.5	dictionary<int8, string>
AccNumber	114.4	9.5	−104.9	dictionary<int8, string>
LocPerilsCovered	104.9	9.5	−95.4	dictionary<int8, string>
LocCurrency	104.9	9.5	−95.4	dictionary<int8, string>
CountryCode	95.4	9.5	−85.9	dictionary<int8, string>
LocNumber	171.7	171.7	0	string (correctly rejected)

int8 is selected here because the benchmark data has ≤128 unique values per column. A portfolio with more unique accounts steps up to int16 (2 bytes/row) or int32 (4 bytes/row) automatically.

closes #190

Updated build and pacakging of ods-tools - (PR #278)

• Fixed build issue in CI by switching from sub-classing build_py instead of install
• Removed the --local-oed-spec flag, logic for download custom spec is moved into build workflow
• Moving to using pyproject.toml over setup.py