Skip to content

Add verify-watermark-removal skill - #320

Merged
davepoon merged 2 commits into
davepoon:mainfrom
Yurakonoplya:add-verify-watermark-removal-skill
Sep 14, 2026
Merged

davepoon merged 2 commits into
davepoon:mainfrom
Yurakonoplya:add-verify-watermark-removal-skill

Conversation

@Yurakonoplya

Copy link
Copy Markdown
Contributor

Summary

Adds verify-watermark-removal, a skill that measures whether a "watermark remover" or "humanizer" actually removed a statistical text watermark, instead of repeating the tool's own claim.

The check is possible because the mark is planted first, under a key the user holds. A detector that knows the key is the strongest detector that can exist for that text, so the score is a ceiling rather than a guess. The skill also reports what the rewrite cost: meaning kept, facts kept, longest verbatim run, share of words changed, length ratio.

It only measures. It removes nothing, and it never claims anything about a specific vendor's watermark.

Component Details

Testing

  • Ran validation (node scripts/validate-skills.js and node scripts/validate-all.js, both pass)
  • Tested functionality: the four CLI commands the skill drives (generate, check, check --json, matrix) are the ones the tool ships, and the example output in the skill is the real output format of unmark-checker check
  • No overlap with existing components: no skill here plants or scores a text watermark

Examples

  1. "I ran this paragraph through a humanizer that promises to strip AI watermarks. Did it work?" The skill generates a keyed sample, asks the user to run the tool on it, scores what came back, and answers mark_present, uncertain or mark_gone with the cost columns.
  2. "Which of these three removal services is worth paying for?" Each run is recorded as a service file, unmark-checker matrix turns the folder into one table, and the services are compared on the same measurements.
  3. "Is my text undetectable now?" The skill refuses that framing: clearing this mark has not been shown to clear any vendor's mark, and it says so instead of rounding a passing run into a promise.

…e key gives a scale rather than hiding the sample, generate warns instead of refusing on samples the mark missed, cost numbers are absent for not_our_text, the vendor-mark carry-over is an inference, the key is read without echo and kept out of the process under test, and the install pins a tag)
@davepoon

Copy link
Copy Markdown
Owner

Thanks for the careful write-up — reviewed for security, prompt injection, and fit with this collection.

Security / prompt injection: The skill markdown is clean (no override/exfil instructions). It only measures; it never removes a mark and refuses the “now undetectable” framing. Key handling is sensible (read -rs, keep it out of files and shell history). Spot-checked unmark-checker at tag v0.1.5: local transformers/torch stack, no outbound HTTP client in the detector/CLI path. Pinning the git tag instead of main is the right call.

Codebase fit: Frontmatter validates (name matches the directory, category: testing-qa, license allowed). No overlap with an existing skill. External CLI dependency is documented the same way other install-from-clone skills are.

Merging.

@davepoon
davepoon merged commit a6c484b into davepoon:main Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants