v3.0.0

Pixels 3.0.0

Pixels 3.0.0 is the GA release of the next-generation Databricks medical-imaging
accelerator. It builds on 3.0.0-rc.1 (and the nine release candidates that followed)
and ships the complete repository reorganization, a fully-declarative Databricks
Asset Bundle (DAB) installer with a 14-task DAG, and hardened Lakebase, DICOMweb
gateway, OHIF viewer, Vista3D model serving, Genie, and Lakeview
dashboard deployment, alongside a long list of stability, performance, and
documentation improvements.

For the feature highlights introduced over 2.2.0 (DICOMweb Gateway, Lakebase
integration, Variant data type, DICOM Redactor, Genie automation, OHIF v3.11.1,
MONAI 1.5.2, etc.), see the v3.0.0-rc.1 release notes.

This changelog covers every change since 3.0.0-rc.1 — 208 commits, 9 release
candidates, 14 merged pull requests.

---

Highlights since 3.0.0-rc.1

• Repository reorganization — Python library moved to src/dbx/pixels/ (src layout),
  apps consolidated under apps/, model assets under models/monai/, install task
  notebooks under install/, demos under notebooks/, docs under docs/.
• One-click DAB installer — a single make deploy + databricks bundle run pixels_install
  provisions the entire stack: UC schema/volume/tables, Lakebase Postgres instance with
  Reverse ETL sync, DICOMweb gateway and OHIF viewer apps, Vista3D model serving endpoint,
  Genie space, Lakeview dashboard, STOW processor job, and grants.
• 14-task install job with explicit dependencies, retry-free execution, and a final
  validate_install task that verifies every surface end-to-end.
• Cross-workspace, cross-catalog ready — every notebook is parameterized via widgets
  fed by databricks.yml variables. Default catalog is now main, schema is pixels.
• Idempotent deployment — every install task can be re-run without breaking the stack
  (schemas, endpoints, apps, Lakebase grants, VS index, Genie space).
• Lakebase robustness — DML grants for app service principals, role-propagation race
  fix, deduped instance_paths_vw to avoid PK violations on Reverse ETL sync, automated
  synced-table creation via API, full connectivity/sync/grants/writes tests in
  validate_install.
• Model serving robustness — split task 03 into 03a register / 03b deploy /
  03c validate; endpoint creation and update are fully idempotent; predict calls use
  the REST API with explicit timeouts; stale-config detection across schema/model
  renames; champion alias on registration.
• OHIF mixed-content fix on HTTPS Databricks Apps origins; viewer-side reverse proxy
  for MONAI info / infer calls (cross-app OBO with request-correlated logging).
• Wheel + OHIF served from UC Volume — deploy_prep uploads databricks_pixels-*.whl
  and ohif.tar.gz to /Volumes/<catalog>/<schema>/<volume>/. Apps pip install the
  wheel from the Volume at startup and extract OHIF in-container, removing large
  .wasm files from DAB sync.
• STOW-RS hardening — gateway derives stow_operations table dynamically from the
  configured Pixels table, normalizes cache keys to composite (study, series, sop)
  instance UIDs, and ships unit tests for the handler.
• Performance — mapInPandas-based parallel unzip for large archives, concurrent
  I/O in DicomMetaExtractor, configurable worker counts.
• Documentation — docs/INSTALL.md rewritten for the 14-task DAG, prerequisites,
  and scale_to_zero_enabled. New CLAUDE.md capturing project structure, build
  commands, conventions, and the mandatory post-install integration-test suite.

---

Repository reorganization (the reorg branch)

The repository was restructured in seven phases to clearly separate library code,
install notebooks, apps, models, and demo material:

• Phase 1 — consolidate install tasks into install/ (7d7f4d6)
• Phase 2 — extract apps to apps/, move shared app code to dbx/pixels/common/ (3dcef06)
• Phase 3 — move the library to src/dbx/pixels/ (src layout); update setup.py (0c29e33)
• Phase 5 — move the model to models/vista3d/, demos to notebooks/, docs to docs/ (4e10863)
• Phase 6 + 7 — style fixes, bundle validation, refresh README.md and CLAUDE.md links (c0080b0)
• Rename models/vista3d/ → models/monai/ to reflect broader scope (MONAI Label + Vista3D bundle) (13e21bd)
• Update CLAUDE.md for the reorg; fix pre-commit paths for the src layout (b50130a)
• Restore OHIF in package resources for wheel distribution (a647b7f)
• Decouple Vista3D model from DAB sync; move plan docs to plans/ (80ba431)

make dev, make build, make deploy, make test, make style, make check, and
make clean continue to work with the new layout. The Asset Bundle is configured to
sync the right subset of the new tree and to include dist/*.whl.

DAB installer & install job

A complete DAB installer replaces ad-hoc deployment notebooks. The install job is a
14-task DAG that provisions everything top-to-bottom.

Structure

• Add resources/install-job.yml, resources/dashboard.yml, resources/unity-catalog.yml DAB resources (9b7e40e)
• Split task 03 into 03a register-model, 03b deploy-endpoint, 03c validate-model (09d46b6)
• Split deploy-apps into 5 install tasks: deploy_prep, deploy_lakebase,
  deploy_gateway, deploy_viewer, deploy_grants (1a0965b)
• Drop numeric task prefixes from task names (1a0965b)
• Add 10_validate install task to verify every Pixels surface (73ea589)
• Add 09-post-install-update task for dashboard params + app thumbnail (3685dec)
• Add stow_processor task — job creation + perms (65d2f76)
• Move init-schema into the DAB; dedup Genie space; fix 03a restart (65d2f76)
• Add install/uninstall.py to tear down a Pixels deployment (2c62bf9)
• Retry validation up to 12× at 3-min intervals until all checks pass (3e39948)
• Disable per-task retries (max_retries: 0) so failures surface immediately (bf645fc)
• Increase serverless compute environment from 1 to 4 (d4f2860)
• Wire lakebase_instance_name through DAB → job → notebook widgets (6e35338)
• Add scale_to_zero_enabled variable; pin deps; reorder job tasks (d1650c0)
• Detect stale serving-endpoint config across schema/model name changes (4bf62ef)
• Skip trashed Pixels dashboards in post_install_update (51ed6cf)
• Split app-shell creation from code deploy to fix fresh-install grants race (9a48438)
• Template dashboard params at make deploy time, not via post-install patch (ce29876)
• Broaden pre-commit E2E test rule to apply to every commit (f770544)
• Address PR #211 review feedback for 3.0.0 release prep (ecbbbd7)
• Update Makefile and requirements.txt for improved environment setup (7cc7041)

Configuration & parameterization

• Parameterize catalog references for cross-workspace installs (22968df)
• Parameterize VLM/LLM names (f9d8e77)
• Parameterize dashboard table, inline app deps, and upgrade CUDA/PyTorch stack (7e90cbb)
• Change default catalog from dmoore to main; update INSTALL.md (a030a9a)
• Init DDL objects; move init_tables out of Catalog.init() (7b77ff5, 247d8fd)
• Fix init_schema notebook to be self-contained on serverless (69a02f3)
• Convert 00-init-schema to pure SQL; fix 10_validate failures (6ecd2f0)
• Fix catalog creation on Default Storage workspaces (9709d6a)
• Handle ABAC policy error in init_env table existence check (e429de0)

App packaging & deployment

• Decouple OHIF from the wheel; serve from the UC Volume at app startup (ae079ac)
• Upload the wheel to the UC Volume for app pip install (8434113)
• Restore OHIF in package resources for wheel distribution (a647b7f)
• Clean up leftover wheel files from app dirs before deploy (0698194)
• Replace git+https dep with the UC Volume wheel in the stow-processor job (3fa1765)
• Use the local wheel for app deps instead of git+https (6857e16)
• Switch to git+https@fix/install during fix-up, then back to the local wheel (633fbe7)
• Fix BUILD cache: version comment in requirements.txt; Makefile ohif.tar.gz (38ea38c)
• Add app-ping.sh smoke test for the DICOMweb viewer app (aa85375)
• Add tests/deployment/pixels_inference.sh model-serving smoke test (e6d3c4c)
• Self-host landing page CSS; fix thumbnail API; improve app idempotency (b1c0727)

Lakebase

• Add Lakebase tests to validate-install: connectivity, sync, grants, writes (510ffd0)
• Fix Lakebase grants: add UPDATE + DELETE on all tables, default privileges (d07eecb)
• Grant Lakebase DML privileges to the app service principal (fd509e8)
• Fix Lakebase role-propagation race in get_or_create_sp_role (2d12f92)
• Deduplicate instance_paths_vw to prevent PK violation on Reverse ETL sync (b59d068)
• Automate Reverse ETL synced-table creation via API (92e5ef7)
• Add synced-table verification, retry for deploy_apps, clarify Reverse ETL docs (6e416e6)
• Refactor endpoint retrieval logic in lakebase.py (6c6ee2a, plus style passes)
• Update hypercorn dependency to include HTTP/2 (h2) support (2181742, 1a309f1)
• Enable Managed File Events config for Lakebase pipelines (f2060ad)

Model serving (Vista3D / MONAI Label)

• Default DICOMWEB_USE_USER_AUTH to true (OBO auth) (4d5047d)
• Fix cross-app OBO and add request-correlated debug logging (610b264)
• Align dcm_ingest databricks-sdk pin with install pipeline (0.60.0 → 0.88.0) (8d9e80c)
• Vista3D conda env: drop cuda-toolkit, bump pydicom, pin pylibjpeg-openjpeg (45c3c33)
• Add champion alias, model description, and fix MONAI GET proxy (f1a80fe)
• Remove vista3d_bundle/vista3d sync exclude to fix model packaging (cb7f6de)
• Pin torch 2.5.1+cu124; align Python 3.12.3; fix 03a...

v3.0.0-rc.9 - Pre-Reorg

What's Changed

• Improved prompt of the genie space by @erinaldidb in #203
• Notebook demonstrating metatag deid by Presidio and Part 15 rules by @yenlow in #169
• Feat/permissive by @dmoore247 in #206

Full Changelog: 3.0.0-rc.8...3.0.0-rc.9

3.0.0-rc.8

What's Changed

• Refactor unzip functionality for improved parallel processing by @erinaldidb in #200

Full Changelog: 3.0.0-rc.7...3.0.0-rc.8

v3.0.0-rc.7

What's Changed

• Update dashboard layout and default values by @erinaldidb in #196
• Enhance DICOM metadata extraction by removing UN VR elements by @erinaldidb in #197

Full Changelog: 3.0.0-rc.6...3.0.0-rc.7

v3.0.0-rc.6

What's Changed

• Enhance DicomMetaExtractor with concurrent I/O by @erinaldidb in #195
• Auto-create STOW processor job and per-table path scoping by @erinaldidb in #193

Full Changelog: 3.0.0-rc.5...3.0.0-rc.6

v3.0.0-rc.5

What's Changed

• Refactor endpoint retrieval logic in lakebase.py by @erinaldidb in #188
• Add file meta header tags to DICOM metadata extraction by @erinaldidb in #189
• Fix DICOM identifier uniqueness in InstancePathCache by @erinaldidb in #192
• Enable managed file events configuration by @erinaldidb in #190
• Refactor STOW handling in DICOM web gateway by @erinaldidb in #191

Full Changelog: 3.0.0-rc.4...3.0.0-rc.5

v3.0.0-rc.4

What's Changed

• Update hypercorn dependency to add HTTP/2 support by @erinaldidb in #187

Full Changelog: 3.0.0-rc.3...3.0.0-rc.4

v3.0.0-rc.3

Pixels 3.0.0 Release Candidate 2

What's Changed - BugFixes

• Refactor modalities and query parameters handling in queries.py by @erinaldidb in #186

Full Changelog: 3.0.0-rc.2...3.0.0-rc.3

v3.0.0-rc.2

Pixels 3.0.0 Release Candidate 2

What's Changed - Bugfixes

• Change staticWado from true to false by @erinaldidb in #184
• Rename lakebase default branch to production and fix endpoint name for gateway metrics by @erinaldidb in #185

Full Changelog: 3.0.0-rc.1...3.0.0-rc.2

v3.0.0-rc.1

Pixels 3.0.0 Release Candidate 1

Highlights

Pixels 3.0.0 is a major release that introduces DICOMweb Gateway, Lakebase integration, Variant Data Type support, a full DICOM Redactor, Genie Space automation, and significant performance and security improvements across the board.

---

What's New

DICOMweb Gateway (Production-Ready)

• Full DICOMweb compliant architecture with split dicom_web (viewer) + dicom_web_gateway (QIDO/WADO/STOW) Databricks Apps
• QIDO-RS: Query studies, series, and instances with full DICOM UID hierarchy support
• WADO-RS: Retrieve DICOM instances and frames with progressive frame streaming, multiframe support, and gzip compression
• STOW-RS: Store DICOM files with streaming upload, background processing jobs, and concurrency control
• HTTP/2 support via Hypercorn for improved throughput
• Built-in benchmark dashboard and metrics dashboard for monitoring gateway performance
• BOT (Basic Offset Table) cache with priority-based prefetching algorithm for fast frame retrieval
• LRU prefetcher cache with configurable cache sizes (up to 4GB)
• Cloud-direct upload path for optimized STOW performance
• Token passthrough, credential caching, and refreshable token pool for robust auth

Lakebase Integration

• Lakebase Autoscale support for tier-2 file path caching and frame indexing
• Unity Catalog-aligned database and schema mapping with Reverse ETL Sync compatibility
• Row-Level Security (RLS) sync with Unity Catalog groups and rules
• Configurable connection pool sizes with refreshable token pools
• SQL injection prevention using sql.Identifier parameterization
• Dedicated SQL scripts for schema creation, DICOM frames tables, metrics, and RLS policies

Variant Data Type Support

• Full compatibility with Databricks Variant Data Type for DICOM metadata storage
• Variant migration script for upgrading existing catalogs
• Updated OHIF viewer and MONAILabel clients for variant-compatible data handling
• NaN record handling and binary image cleanup for JSON conversion

DICOM Redactor

• New Redactor class for PHI redaction in DICOM metadata and pixel data
• LLM-powered metadata redaction with configurable prompts
• Bbox-based pixel redaction without OpenCV dependency (uses plain arrays)
• Support for single and multi-layer JPEG2000 lossy/lossless compression
• Redaction job API with notebook-driven task execution
• Comprehensive unit tests for the redactor pipeline

Genie Space Automation

• Programmatic Genie Space creation via API
• Vector Search integration with DICOM tags index
• Automated setup notebook for Genie Spaces

OHIF Viewer Upgrade

• Upgraded to OHIF v3.11.1 (from 3.9.x) — major viewer upgrade
• Added microscopy viewer support
• Better multiframe DICOM handling with HTJ2K encoding
• OOM fix for HTJ2K DICOMs in browser
• OIDC compatibility for DICOMweb integration
• MONAI Label button restored in viewer
• Volume Rendering preset images added

MONAILabel & Model Serving

• Upgraded to MONAI 1.5.2 and MONAILabel latest
• Upgraded to PYDICOM 3.0.1
• Migrated models to generate .dcm segmentation files (DICOM SEG)
• Fix on writing compressed DCM SEG files
• GPU loading and request handling configuration
• Custom NibabelWriter for improved segmentation output
• DBR version upgraded to 17.3.x-scala2.13

Databricks Asset Bundles (DABs)

• Full DABs support for demo deployment (dabs/dbdemos/)
• Split resources into independent YAML files (dashboard, ingest, viewer app, Unity Catalog)
• Notebook-driven DAB installation (99-run-install.ipynb)

VLM Integration

• Vision-Language Model integration for DICOM image analysis
• Configurable prompts with temperature and max_tokens settings
• Prompt handler class for structured VLM interactions

---

Performance Improvements

• Async WADO-RS with dedicated I/O pool and increased parallelism
• Streaming responses for metadata and frame retrieval
• SQL connection pooling with batched inserts
• Pre-cached instance paths on startup
• One-shot indexer for uncompressed multiframe DICOMs
• fsspec-based remote zip extraction for huge files
• Configurable sequential/concurrent upload modes
• Memory limits for queue uploads

---

Security Fixes

• MODERATE-02: Internal error details no longer leaked in HTTP responses
• MODERATE-04: Gateway request counters made thread-safe
• MODERATE-05: assert replaced with proper validation in production code
• MODERATE-07: Potential PHI removed from debug logs
• MODERATE-12: Credential stampede guard logic bug fixed
• LOW-01: Inconsistent databricks-sdk version pinning resolved
• LOW-03: Token prefix cache key collision risk mitigated
• LOW-05: File handle leaks in middleware fixed
• LOW-07: Unmounted dicomweb_perf_compare function fixed
• LOW-08: Duplicate GZipMiddleware import removed
• SQL injection prevention in Lakebase queries
• Volume existence check logic corrected

---

Infrastructure & CI

• New CI workflows: safe PR checks, manual lint/test runner, manual AWS integration tests
• Separated requirements.txt and requirements-ai.txt for ML/non-ML use cases
• Updated Makefile with venv support
• Notebooks migrated to .ipynb format (01-dcm-demo, workflows)
• Added DatabricksFile class for Volume file validation and extraction
• pylibjpeg-libjpeg made optional (instructions in README for codec setup)

---

Breaking Changes

• OHIF v3.11.1 includes breaking changes from v3.9.x — custom extensions may need updates
• Lakebase now uses PIXELS schema instead of public
• Deep parameter defaults to False in catalog operations
• init_tables removed from Catalog.init() — use fixtures instead
• Workflow scripts migrated from .py to .ipynb

---

Dependencies

• databricks-sdk updated and version-pinned consistently
• psycopg2 upgrade for Lakebase connectivity
• python-multipart added for STOW-RS support
• pydicom 3.0.1
• MONAI 1.5.2
• Removed cv2 as hard dependency (optional for frame handling)

---

Full Changelog: 2.2.0...3.0.0-rc.1