3.1.1

[3.1.1] - 2025-02-24

Highlights:

• sample: is now a "smart" command that uses the stats cache to validate and make sampling faster.
• With the QSV_STATSCACHE_MODE env var, you can now control the stats cache behavior suite-wide, making sure "smart" commands use it when appropriate.
• luau command's capabilities have been significantly expanded with:
  • New accumulate helper function for aggregating values across rows
  • Optional naming for cumulative helper functions
  • More robust error handling and improved docstrings
  • Enhanced scripting performance with fast-float parsing
  • new Wiki section with examples of using its helper functions
• schema: now does type-aware sorting of enum lists, making JSON Schema enum list customization easier when fine-tuning it for JSON Schema validation with validate.
• lens: adds --freeze-columns option with a default of 1, improving navigation of wide CSVs
• stats: adds --dataset-stats option to explicitly compute dataset-level statistics. Starting with qsv 2.0.0, it was computed automatically to support Datapusher+ and the DRUF workflow, but it was causing confusion with some command-line users.

---

Added

• lens: added --freeze-columns option #2552
• luau: added accumulate helper function #2537 #2539
• luau: added a new section in the Wiki with examples of using the new helper functions https://github.com/dathere/qsv/wiki/Luau-Helper-Functions-Examples
• sample: is now "smart" - using the stats cache to validate and make sampling faster #2529 #2530 71ec7ed
• schema: added type-aware sort of JSON Schema enum list #2551
• stats: added --dataset-stats option #2555
• python: added precompiled qsvpy binary for Python 3.13 c408778
• added QSV_STATSCACHE_MODE env var to control stats cache suite-wide 4afb98d 2adc313 ba75f08
• docs: updated PERFORMANCE docs and added a TLDR version 77ed167 c61c249 db0bb3f
• chore: added *.tab & *.ssv to typos config 5236675

Changed

• frequency: made error handling more robust b195519
• luau: refactored all cumulative helper functions (cum_) now have name as an optional argument #2540
• schema: refactored to use QSV_STATSCACHE_MODE env var 5771ff4
• select: refactored select helper bfbe64c
• stats: optimized memory layout of central Stats struct 52f697e
• stats: optimized record_count functionality 0e3114a 18791da
• contrib(completions): update qsv completions for qsv 3.1 by @rzmk in #2556
• deps: bump arrow and tempfile 4cc2679
• deps: bump cached and redis crates e622d14
• deps: bump csvlens from 0.11 to 0.12 b2fd985
• deps: use our patched fork of csvlens with ability to freeze columns d66ec6d
• deps: bump polars to 0.46.0 at py-1.23.0 tag 6072aa2
• deps: bump flate2 from 1.0.35 to 1.1.0 eed471a
• deps: bump gzp from 0.11 to 1.0.0 43c8a4a
• build(deps): bump jaq-json from 1.1.0 to 1.1.1 by @dependabot in #2547
• build(deps): bump jaq-core from 2.1.0 to 2.1.1 by @dependabot in #2546
• build(deps): bump log from 0.4.25 to 0.4.26 by @dependabot in #2545
• build(deps): bump tempfile from 3.16.0 to 3.17.0 by @dependabot in #2532
• build(deps): bump tempfile from 3.17.0 to 3.17.1 by @dependabot in #2535
• build(deps): bump serde_json from 1.0.138 to 1.0.139 by @dependabot in #2541
• build(deps): bump serde from 1.0.217 to 1.0.218 by @dependabot in #2542
• build(deps): bump smallvec from 1.13.2 to 1.14.0 by @dependabot in #2528
• build(deps): bump strum from 0.27.0 to 0.27.1 by @dependabot in #2533
• build(deps): bump strum_macros from 0.27.0 to 0.27.1 by @dependabot in #2534
• build(deps): bump uuid from 1.13.1 to 1.13.2 by @dependabot in #2538
• build(deps): bump uuid from 1.13.2 to 1.14.0 by @dependabot in #2544
• chore: we now have ~1,800 tests! f5d09ed
• applied select clippy lint suggestions
• bumped indirect dependencies to latest versions
• bumped MSRV to latest Rust stable - v1.85

Fixed

• count: refactored to fall back to "regular" CSV reader when Polars counting returns a zero count fd39bcb
• schema: fixed off-by-one error 60de090
• ensured get_stats_record helper returns field/stats correctly ad86a37
• Fixed RUSTSEC-2025-0007: ring is unmaintained #2548
• stats: only add qsv__value column when --dataset-stats is enabled 64267d3
• skip format check when path starts with temp dir (indicating its a file streamed from STDIN) or is a snappy file ff8957e

Removed

• frequency: removed --stats-mode option now that we have a suite-wide QSV_STATSCACHE_MODE env var ba75f08 416abb7
• chore: removed simdutf8 conditional directive for aarch64 architecture, now that its no longer needed ec1e16c
• removed publish-linux-qsvpy-glibc-231-musl-123.yml workflow as it was getting cross compilation errors and we have another musl workflow that works 7c08617

Full Changelog: 3.0.0...3.1.1