Skip to content

DEVELOPING.md

Latest commit

 

History

History
187 lines (135 loc) · 6.39 KB

DEVELOPING.md

File metadata and controls

187 lines (135 loc) · 6.39 KB
Raw

Developing cpex-plugins

Repository Model

This repository currently manages one plugin class: Rust plugins that are built with PyO3/maturin and published to PyPI as Python packages.

Managed plugin path:

plugins/rust/python-package/<slug>/

Every managed plugin must satisfy the catalog contract enforced by tools/plugin_catalog.py:

  • distribution name: cpex-<slug>
  • Python module: cpex_<slug>
  • Cargo.toml is the version source of truth
  • cpex_<slug>/plugin-manifest.yaml version matches Cargo.toml
  • cpex_<slug>/plugin-manifest.yaml defines top-level kind in module.object form
  • pyproject.toml publishes the matching plugin class reference under [project.entry-points."cpex.plugins"] in module:object form
  • plugin Cargo.toml repository metadata points to https://github.com/IBM/cpex-plugins
  • plugin crate is listed in the top-level workspace Cargo.toml

Working on One Plugin

cd plugins/rust/python-package/rate_limiter
uv sync --dev
make install
make test-all

Swap rate_limiter for any other managed plugin slug.

Secrets Detection Count Semantics

secrets_detection reports one finding per non-overlapping secret span. When multiple enabled patterns match the same bytes, or overlapping bytes, the scanner redacts the merged span once and reports the most specific matching detector type. Distinct non-overlapping secrets in the same payload still count separately.

This changed older behavior that could count overlapping broad and specific pattern matches as multiple findings. Operators using min_findings_to_block values greater than 1 should audit thresholds when upgrading.

Repo-Level Commands

make plugins-list
make plugins-validate
make plugin-test PLUGIN=pii_filter

make plugins-validate runs the same convention checks that the repo contract CI workflow runs. It runs the catalog validator plus the shared repo contract test modules: tests/test_plugin_catalog.py and tests/test_install_built_wheel.py.

Adding a New Managed Plugin

Using the Plugin Scaffold Generator (Recommended)

The easiest way to create a new plugin is using the scaffold generator:

make plugin-scaffold

This interactive tool will:

  • Prompt for plugin name, description, author, and version
  • Let you select from 12 available hooks across 5 categories
  • Generate complete plugin structure with all required files
  • Create comprehensive unit tests (Python and Rust)
  • Set up build configuration and documentation

For non-interactive mode:

python3 tools/scaffold_plugin.py --non-interactive \
  --name my_plugin \
  --description "My plugin description" \
  --author "Your Name" \
  --hooks prompt_pre_fetch,tool_pre_invoke

After scaffolding:

  1. Review and customize the generated code in plugins/rust/python-package/<slug>/
  2. The crate is automatically added to the workspace Cargo.toml
  3. Run make plugins-validate to verify structure
  4. Run make plugin-test PLUGIN=<slug> to execute the plugin's full make ci flow

Manual Plugin Creation

If you prefer to create a plugin manually:

  1. Create plugins/rust/python-package/<slug>/.
  2. Add the required files and package/module names that match the slug conventions.
  3. Add the crate path to the workspace members list in the top-level Cargo.toml.
  4. Run make plugins-validate.
  5. Run make plugin-test PLUGIN=<slug> to execute the plugin's full make ci flow.

Releasing

Releases are per plugin and version-bump driven. Use this process to publish a new version of an existing managed plugin to PyPI.

  1. Pick the plugin slug and new version.

    The plugin slug is the directory name under plugins/rust/python-package/<slug>/, for example rate_limiter. The tag slug is the hyphenated form, for example rate-limiter.

  2. Update the version files.

    Cargo.toml is the version source of truth. The plugin manifest and top-level lockfile must stay consistent with it.

    $EDITOR plugins/rust/python-package/rate_limiter/Cargo.toml
$EDITOR plugins/rust/python-package/rate_limiter/cpex_rate_limiter/plugin-manifest.yaml
cargo update -p rate_limiter --precise 0.0.5

  3. Run local validation.

    make plugins-validate
make plugin-test PLUGIN=rate_limiter

  4. Merge the version bump to main.

  5. Let CI create the release tag and publish.

    On a main push, .github/workflows/ci-rust-python-package.yaml detects plugin Cargo.toml version bumps. After the build, security, coverage, and documentation jobs are green, it creates the release tag at the merge commit and invokes .github/workflows/release-rust-python-package.yaml with PyPI publishing enabled.

    The workflow uses GITHUB_TOKEN to push release tags. Repository tag protection or rulesets for release tag patterns must allow that token, or the workflow must be updated to use an approved GitHub App or PAT token.

    Release tags use the hyphenated plugin slug, not the directory/module underscore form. CI creates tags in this form:

    rate-limiter-v0.0.5

    Examples:

    • rate_limiter -> rate-limiter-v0.0.5
    • secrets_detection -> secrets-detection-v0.2.2

    Use make plugins-list to inspect the current managed plugin slugs and package names. Do not create the tag manually for ordinary releases; manual tag pushes are reserved for recovery or explicit release-maintainer action.

  6. Watch the release workflow and confirm publish success.

    gh run list --workflow ci-rust-python-package.yaml --branch main --limit 5
gh run list --workflow release-rust-python-package.yaml --limit 5
gh run watch <run-id> --exit-status

  7. Verify the package exists on PyPI at the new version.

    uv run python -m pip index versions cpex-rate-limiter

    The release page should also exist at https://pypi.org/project/cpex-rate-limiter/0.0.5/.

The CI workflow creates tags only after the required checks pass. It then calls the release workflow directly for publishing; it does not rely on a bot-created tag push to start another workflow run. The release workflow resolves the tag back to the managed plugin path, validates metadata and versions, then builds and publishes only that plugin. PyPI publishing is allowed only for release tags that point at main.

Dependency refresh work is separate from the release process. Track broader dependency or ContextForge updates outside a plugin release PR.