Phase 1.2: Build Tooling Modernization - #2141
Merged
Merged
Conversation
Add Phase 1.2 Completion Summary with GitHub issues and PRs for all 6 repos: - txaio: Issue crossbario#202, PR crossbario#203 - autobahn-python: Issue crossbario#1787, PR crossbario#1788 - zlmdb: Issue crossbario#79, PR crossbario#80 - cfxdb: Issue crossbario#102, PR crossbario#103 - wamp-xbr: Issue crossbario#154, PR crossbario#155 - crossbar: Issue crossbario#2140, PR crossbario#2141 All repos using branch modernization-phase-1.2. Note: This work was completed with AI assistance (Claude Code).
Added comprehensive appendix documenting: - WAMP ecosystem dependency architecture (txaio → autobahn → zlmdb/xbr → cfxdb → crossbar) - The critical `install-dev-local` recipe for cross-repo development - asgard1 infrastructure design for sibling repo layout - pyproject.toml dependency configuration patterns - Build environment configuration for PyPy compatibility - Cross-repo development workflow examples - Key design patterns for dependency ordering and extras This analysis captures the sophisticated infrastructure that enables simultaneous development across the entire WAMP Python ecosystem. Note: This work was completed with AI assistance (Claude Code).
- Rename setup.cfg to setup.cfg.orig (pycodestyle config replaced by ruff) - Rename tox.ini to tox.ini.orig (replaced by comprehensive justfile) - Fix PEP 639 license format: use SPDX expression "EUPL-1.2" instead of deprecated TOML table format - Remove License classifier (superseded by license expression per PEP 639) Build tooling now fully modernized: - pyproject.toml: PEP 621 metadata, ruff, mypy, pytest, coverage configs - justfile: 50+ recipes for all development tasks - setup.py: Minimal shim for env vars (LMDB_FORCE_CFFI, etc.) Verified: - ruff format/lint: PASSED - bandit security: PASSED (no medium/high severity issues) - twine check: PASSED - wheel build: crossbar-25.11.1-py3-none-any.whl Note: mypy reports 26 type errors in existing code - these are known issues (disallow_untyped_defs=false) and not blocking for Phase 1.2. Note: This work was completed with AI assistance (Claude Code).
…r analysis - Add branded header banner to justfile default recipe (consistent with other WAMP ecosystem projects) - Update Phase 1.2 task checklist to show completion status for all repositories - Add completion summary table showing all 6 repos complete - Add Appendix: wamp-xbr - Dual Build System Analysis documenting: - Python + Solidity/Truffle dual build architecture - Symlink-based artifact bundling (xbr/abi → build/contracts) - Justfile Makefile wrapper recipes - Ethereum ecosystem dependencies (Web3.py, eth-abi, eth-account) - Build order dependencies (Solidity compile before Python install) Note: This work was completed with AI assistance (Claude Code).
Add new MODERNIZATION.md appendix section documenting: - Overview statistics (160+ transitive deps, 25 native packages) - Full dependency tree diagram for WAMP ecosystem core - Package classification by groups: - Group 1: WAMP Ecosystem (own packages) - all pure Python - Group 2: Cryptography/Security (CFFI-based, PyPy compatible) - Group 3: Serialization (mixed native with fallbacks) - Group 4: Twisted Ecosystem - Group 5: Ethereum/XBR Stack - Group 6: HTTP/Async Stack - Group 7: Core Utilities with Native Extensions (CPyExt) - Group 8: Pure Python Utilities - CFFI vs CPyExt classification for all native packages - PyPy compatibility matrix with environment variables - Wheel availability matrix (x86_64, ARM64, CPython, PyPy) - Full alphabetical dependency list with type annotations - Recommendations for PyPy, RHEL9, ARM64, and security scanning This analysis supports Phase 1.3 (Wheel Building) planning and enterprise deployment decisions. Note: This work was completed with AI assistance (Claude Code).
…ages Correct the dependencies analysis to properly reflect that autobahn and zlmdb are NOT pure Python packages - they include CFFI-based native extensions with binary wheels: - autobahn: _nvx_utf8validator, _nvx_xormasker (WebSocket accelerators) - zlmdb: _lmdb_cffi (LMDB database bindings) These CFFI extensions are a deliberate design choice for PyPy compatibility while achieving near-native performance. Updates: - Group 1 table: mark autobahn/zlmdb as "Native (CFFI)" with binary wheels - Dependency tree: add [NATIVE:CFFI] markers - CFFI summary: add autobahn and zlmdb to list - Statistics: update CFFI count from ~5 to ~7 - Full list: add NATIVE:CFFI/Pure Python markers to WAMP packages Note: This work was completed with AI assistance (Claude Code).
Add critical documentation of the WAMP ecosystem vendoring strategy: ### Vendored Packages - LMDB: Vendored into zlmdb with CFFI bindings (zlmdb/lmdb/_lmdb_cffi.so) - NOT py-lmdb from PyPI (which uses CPyExt) - Ensures PyPy compatibility and consistent API - FlatBuffers: Vendored into autobahn as top-level package - Bundled in autobahn wheel via pyproject.toml include - Also vendored in zlmdb for reflection schemas ### Known Issue Identified⚠️ crossbar's pyproject.toml has "lmdb>=1.4.0" as direct dependency which is WRONG/REDUNDANT because: - Crossbar imports zlmdb, never lmdb directly - zlmdb vendors its own CFFI LMDB bindings - Having lmdb dep may install py-lmdb (CPyExt) conflicting with design Action: Remove "lmdb>=1.4.0" from crossbar's dependencies. Updates: - Added "Critical: Vendored Dependencies Architecture" section - Updated dependency tree to show [VENDORED] markers - Flagged lmdb in full list as redundant Note: This work was completed with AI assistance (Claude Code).
- Remove "lmdb>=1.4.0" from dependencies - crossbar imports zlmdb, never lmdb directly - Add comment to prevent future re-introduction of this dependency - Update MODERNIZATION.md to mark the issue as resolved - Remove lmdb from Group 7 dependencies list zlmdb vendors its own CFFI LMDB bindings at zlmdb/lmdb/_lmdb_cffi.so, which is crucial for PyPy compatibility. Having a direct lmdb dependency could install py-lmdb (CPyExt) which would conflict with this design. Note: This work was completed with AI assistance (Claude Code).
…ecipes spec Phase restructuring: - Phase 1.2: Per-Repo Infrastructure (single branch: modernization-phase-1.2) - Phase 1.2.1: Build Tooling (previously Phase 1.2) - COMPLETE - Phase 1.2.2: Wheel Building (previously Phase 1.3) - Phase 1.2.3: Documentation (previously Phase 1.4) - Phase 1.2.4: Unit Test Coverage (previously Phase 1.5) - Phase 1.3: CI/CD (previously Phase 1.6, new branch: modernization-phase-1.3) Added Common Justfile Recipes Specification: - Environment Management: create, create-all, list-all, version, version-all - Installation: install, install-dev, install-tools, upgrade (+ -all variants) - Quality Checks: check, check-format, check-typing, check-coverage, fix-format - Build/Package: build, build-all, build-sourcedist, verify-wheels, clean-build - Testing: test, test-all - Documentation: docs, docs-clean, docs-view - Publishing: publish, publish-pypi, publish-rtd, download-github-release - Cleanup/Utility: default, distclean, setup-completion This reduces branch churn by keeping related per-repo infrastructure work on a single branch, while CI/CD (which requires GitHub-side configuration) gets its own dedicated branch. Note: This work was completed with AI assistance (Claude Code).
- Rename autoformat → fix-format (keep alias for backward compatibility) - Add check-coverage and coverage alias for test coverage reports - Add upgrade and upgrade-all for dependency management - Add download-github-release for fetching release artifacts - Add publish-pypi workflow (download from GitHub → verify → upload) - Add publish-rtd placeholder for Read the Docs builds Part of Phase 1.2 Build Tooling Modernization to standardize justfile recipes across all WAMP Python ecosystem projects. Note: This work was completed with AI assistance (Claude Code).
Rename deploy-wheels.yml to deploy-wheels.yml.orig to disable the legacy AWS S3/self-hosted runner wheel deployment workflow. The modern release.yml workflow now handles: - PyPI trusted publishing (OIDC) - GitHub Releases (development and production) - Proper wheel and source distribution building This consolidates crossbar's release workflow to a single file (release.yml) matching the pattern used across all WAMP ecosystem packages. Part of Phase 1.2.2 Wheel Building modernization. Note: This work was completed with AI assistance (Claude Code).
- All repositories verified for wheel building - Pure Python packages: py3-none-any wheels (txaio, cfxdb, wamp-xbr, crossbar) - CFFI packages: Platform-specific wheels with auditwheel (autobahn-python, zlmdb) - Standardized on single release.yml per repo - Retired legacy deploy workflows (renamed to .orig) - PyPI trusted publishing (OIDC) configured across all repos - Multi-platform support verified (Linux x86_64, macOS ARM64, Windows) Note: This work was completed with AI assistance (Claude Code).
- Add build-universe: builds wheels for all 6 WAMP Python repos - Add verify-universe: verifies all wheels with twine/auditwheel - Add dist-universe to .gitignore - Fix install-tools to use [dev] instead of [dev,dev-latest] - Add auditwheel to install-build-tools Note: This work was completed with AI assistance (Claude Code).
Comprehensive documentation modernization standards including: - Standardized project directory structure (src layout for all 6 repos) - PEP compliance table (517, 518, 621, 660, 561, 639) - Migration guide from flat layout to src layout - Legacy files to remove (setup.py, setup.cfg, tox.ini, etc.) - Dependency specification policy (libraries vs application) - Production Dockerfile with uv sync --frozen --require-hashes - Complete docs/conf.py template with: - Furo theme with Noto fonts - MyST Markdown configuration - sphinx-autoapi (no-import API docs) - Intersphinx cross-referencing for WAMP ecosystem - sphinx-design, sphinx-copybutton, OpenGraph - Spelling configuration - Per-repo intersphinx mapping table - Documentation dependencies in separate [docs] extra - Standard .readthedocs.yaml template Note: This work was completed with AI assistance (Claude Code).
Add comprehensive overview section from autobahn-python/OVERVIEW.md explaining the group of WAMP projects and their interrelations: - WAMP protocol as umbrella project - Client libraries (Autobahn|Python, JS, Java, C++) - Router (Crossbar.io) - Supporting libraries (txaio, zLMDB, cfxdb) - Python runtime requirements (CPython, PyPy, CFFI) This provides essential context for understanding the modernization effort and why consistency across projects matters. Note: This work was completed with AI assistance (Claude Code).
- Move crossbar/ to src/crossbar/ for proper namespace isolation - Update pyproject.toml: where=["src"], testpaths, coverage source - Update justfile: mypy, bandit, coverage paths to src/crossbar/ - Update docs/conf.py: add sys.path for src layout - Update .ai submodule to 3d10240, add docs/ai symlink - Bump version to 25.12.1 - Update zlmdb>=25.12.1, cfxdb>=25.12.1 dependencies This follows the same pattern as zlmdb, cfxdb, and wamp-xbr migrations. Import test passes: crossbar.__version__ = 25.12.1 Note: crossbar version command requires release key (crossbar-25-12.pub) which will be generated during the release process. Note: This work was completed with AI assistance (Claude Code).
Add a call to `just create-all` before `just build-all` in the build-universe recipe to ensure all Python virtual environments exist before attempting to build wheels. This fixes build failures when venvs haven't been created yet on a fresh clone. Note: This work was completed with AI assistance (Claude Code).
Sync crossbar version with other WAMP ecosystem packages for the December 2025 release cycle. Note: This work was completed with AI assistance (Claude Code).
…e data - Add test-universe-crossbar-version just recipe to verify complete wheel packaging workflow (build, verify, install, run) - Add package-data configuration to include release keys (*.pub) and templates in wheels - Update MANIFEST.in paths for src layout Note: This work was completed with AI assistance (Claude Code).
Contributor
Author
|
progress ... 
|
- Rename test-universe-crossbar-version to install-universe (setup only) - Add test-universe-crossbar-version that depends on install-universe - Add test-universe-crossbar-legal for testing crossbar legal - Add test-universe-crossbar-init for testing crossbar init - Add test-universe-crossbar-start for testing crossbar start (uses autobahn-python example router with configurable timeout) Note: This work was completed with AI assistance (Claude Code).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix #2140
Related Issue(s)
Closes or relates to #2140
Checklist
the style guidelines of this project
is effective or that my feature works
updated the changelog
in this PR