[pull] main from Stirling-Tools:main - #307
Merged
Merged
Conversation
…to the AI (#7580) Split out of #7574 — this is the classification half, which is independent of the editor-source work and can land on its own. ## What this does - **Runs the local heuristic first and only escalates an unsure verdict to the AI.** A high-confidence local answer stands; anything less (or a file the heuristic hasn't reached yet) goes to the engine. A wrong label costs more than an engine call, so the bar is deliberately strict. - **Makes `classify` an authorable pipeline task**, so it can be used as a step like any other tool, and skips files that are already classified. - **Leaves the seeded Classification policy unowned** rather than naming a `system` placeholder that was never a real user; existing seeds are repaired on boot. ## Review feedback applied From @jbrunton96 on #7574: - **The generic runner no longer names classification.** Everything classification-specific moved into `proprietary/data/classificationPolicy.ts`, and `usePolicyAutoRun` now asks capability questions instead: `policyRewritesDocument`, `policyDeliversOutputFiles`, `policyRequiresAiEngine`, `shouldDispatchToAi`. There is no `id === "classification"` left in the runner. - **Ordering is no longer a name in the runner.** `pinClassificationLast` is gone; the runner sorts annotating policies after rewriting ones. The constraint is real: an annotating policy is non-blocking, so a rewriting one running after it forks from the pre-annotation version and drops the labels. To be straight about what this is and isn't - see "Still open" below - `policyRewritesDocument` is still keyed on the category id, not on a property each policy declares. The check moved out of the runner; it did not stop being a check on one id. - **Confidence is typed.** New `ClassificationConfidence` union in `core/types/fileContext.ts`, reused by `fileStorage`, `HeuristicConfidence`, and the trusted-verdict constant instead of being respelled at each site. - **Comments trimmed** to the repo's 2-line guideline, and a stale seeder javadoc that still claimed an internal-user owner was corrected. ## Still open, deliberately `classificationPolicy.ts` answers its capability questions with `categoryId === "classification"`. That is the same check relocated, not removed, and the module doc now says so outright. Deliberate, for two reasons: - **The concept it would be declared against is going away.** Policies are becoming pipelines with labels behind a separate enforcement layer, which removes the category the flag would live on. A capability system built on `categoryId` today gets migrated twice. - **Classification is genuinely privileged, not accidentally special.** It is the only policy with a browser-side implementation, so it can answer without the server. That is a product decision, and a local-only mode for set scenarios is planned - the flag for it should be designed with that feature, not guessed at now. The end state for the rest: an in-place output mode retires the ordering rule and `policyDeliversOutputFiles`, and a run result that can carry findings as well as files retires the remainder. Both touch the import path, which is the most delicate code in `usePolicyAutoRun` - not something to bolt on to a PR that has already been split once. Nothing is broken by leaving it. A user-built classify pipeline still gets its labels: the generic import path reads them off the returned PDF. It versions the file instead of labelling in place, and it misses the local-heuristic shortcut, so it always bills the engine. ## Testing - `classificationPolicy.test.ts` — 12 cases covering each capability and the escalation rule - Full frontend `proprietary` project: 39 files / 442 tests - `:proprietary:test` for `DefaultClassificationPolicySeederTest` + `ClassifyLabelControllerTest` - `tsc --noEmit` on core, proprietary, portal, saas, desktop, cloud --------- Co-authored-by: James Brunton <jbrunton96@gmail.com>
# Description of Changes The Documents tab in the Processor is supposed to be available to all Processor users, but because the API is built on top of the Audit data, which is only for enterprise users, the API call always fails with 403. This means that it never fills the query cache, so every time you go back to the tab it has to reload all the data for a couple of seconds (and will fail again). This fixes the API so that it's available to any Processor user instead of just enterprise users. Also, the documents data was only being written to the log on an enterprise license, so I've changed it so that data is always tracked in the audit log because otherwise the Documents tab would still be useless to non-enterprise users. The Audit Log tab was also available to all Processor users, but would have the same issue where the table would never load because the API would 403 as well. I've just made the Audit Log tab disabled for non-enterprise users now. We might want to do something to signpost it a bit more that it's an enterprise-specific feature, but it's better than nothing for now.
# Description of Changes Stirling engine docker slimming Exclude Python virtualenvs from the Docker build context Drop unused provider SDKs from the engine dependency set Retry the SQLite WAL switch when workers race on startup Build the engine image in two stages and run it unprivileged Swap voyage SDK for api call removing 200MB bloat Bundle the AI engine in the fat image Publish the AI engine as a standalone image 886MB to 295MB in docker file And Docker fat is only 230MB bigger after adding (since it already has python and some deps) --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md) (if applicable) - [ ] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### Translations (if applicable) - [ ] I ran [`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [ ] I have run `task check` to verify linters, typechecks, and tests pass - [ ] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing) for more details.
…7623) Bumps [step-security/harden-runner](https://github.com/step-security/harden-runner) from 2.20.0 to 2.21.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/step-security/harden-runner/releases">step-security/harden-runner's releases</a>.</em></p> <blockquote> <h2>v2.21.0</h2> <h2>What's Changed</h2> <ul> <li>Support for denied endpoints in block mode. This is included in the enterprise tier. Customers can deny outbound calls, for example, to public package registries.</li> <li>Improved Support for AWS CodeBuild GitHub Actions Runners.</li> <li>Bug fixes.</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/step-security/harden-runner/compare/v2.20.1...v2.21.0">https://github.com/step-security/harden-runner/compare/v2.20.1...v2.21.0</a></p> <h2>v2.20.1</h2> <h2>What's Changed</h2> <ul> <li>AWS CodeBuild-hosted runner support</li> <li>Implicitly allow single-labeled (internal) domains in block-mode</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/step-security/harden-runner/compare/v2.20.0...v2.20.1">https://github.com/step-security/harden-runner/compare/v2.20.0...v2.20.1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/step-security/harden-runner/commit/05e31511f85b41b11d1cf0ef85d0992719546e2c"><code>05e3151</code></a> Merge pull request <a href="https://redirect.github.com/step-security/harden-runner/issues/684">#684</a> from step-security/rc-42</li> <li><a href="https://github.com/step-security/harden-runner/commit/0f37afa338f57c61ee3dfc274daca8834963d83e"><code>0f37afa</code></a> fix: ignore denied-endpoints on non-enterprise tier</li> <li><a href="https://github.com/step-security/harden-runner/commit/93b58ee491c5b6cf3a5324966fca2908f8d447f3"><code>93b58ee</code></a> fix: resolve cache host read-first and never downgrade egress policy</li> <li><a href="https://github.com/step-security/harden-runner/commit/e7399dd3e93d6c159d314af54b4704bc48abf6bc"><code>e7399dd</code></a> fix: align deny-list mode detection with agent and log when both endpoint inp...</li> <li><a href="https://github.com/step-security/harden-runner/commit/c16689f716a10cdfd9cfe22e63938b8c6c0657de"><code>c16689f</code></a> test: add denied_endpoints to Configuration fixtures and cover deny-list merge</li> <li><a href="https://github.com/step-security/harden-runner/commit/40b99cf0c7161e4dcdc6c5508927188b65028df9"><code>40b99cf</code></a> Merge pull request <a href="https://redirect.github.com/step-security/harden-runner/issues/682">#682</a> from rohan-stepsecurity/rp/feat/codebuild-self-v2</li> <li><a href="https://github.com/step-security/harden-runner/commit/fedec027a205365a7d64001a81931e4c36a1af6e"><code>fedec02</code></a> Merge branch 'rc-42' into rp/feat/codebuild-self-v2</li> <li><a href="https://github.com/step-security/harden-runner/commit/5361fb178b926b2be6df52e11ee257823821567b"><code>5361fb1</code></a> feat: add build artifacts</li> <li><a href="https://github.com/step-security/harden-runner/commit/286474fffe0b8fe7c9db855f132d04a9b48ab564"><code>286474f</code></a> feat: Support Bravo agent install on CodeBuild runners</li> <li><a href="https://github.com/step-security/harden-runner/commit/051ec05283d064bd82f41279db4f70f0717bf778"><code>051ec05</code></a> Merge pull request <a href="https://redirect.github.com/step-security/harden-runner/issues/683">#683</a> from h0x0er/jatin/deny-list</li> <li>Additional commits viewable in <a href="https://github.com/step-security/harden-runner/compare/v2.20.0...05e31511f85b41b11d1cf0ef85d0992719546e2c">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
github-actions
Bot
added
Documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )