Hello Mr. Chen,
First of all I want to thank you for your work on this plugin. Without it, I would not be writing this message to you now, nor would I have found some of the passions I am deeply obsessed with currently. I currently am building my own solo business in the realm of SEO, AEO and GEO which was born from a conversation I had with my mother about her brick and mortar business and her Shopify storefront. At some point during the journey I stumbled upon your humanizer plugin.
The culmination of that moment is what I'd like to convey to you here in this open issue. I ran a blind study with your tool in conjunction with my own version in great detail. If you have time, I'd love for you to dig into my project in detail. It was initially meant at some level to mask AI assisted prose, but eventually what was born from it is far more important I believe. I'll be including the details below. Again, thank you. I want you to know that this stranger out in the world wide web was truly inspired due to your work. The thing I found is that your pass is the only part of the whole study that had tangible evidence of a clear-cut win.
What worked
Blind pairwise, quality only, authorship never mentioned to the judges:
| contrast |
preferred |
n |
| raw model output vs your rewrite pass alone |
16 / 16 |
16 |
| raw model output vs pass + my checker |
22 / 22 |
22 |
| your pass vs pass + my checker |
12 of 21 (chance) |
21 |
The third row is what I believe is something worth your time. My checker adds nothing measurable on top of your
pass. It emitted its input byte-identical in 11 of 18 cells, so twelve of those twenty-one
trials were judges comparing a file to itself.
Your rewrite pass carries the entire quantitative benefit. It is the sole component in the entire
study with an unambiguous positive result. I built the others on top of it.
What didn't work, which I believe matters for your README
I originally vendored this to try to beat AI detection. It does not, and I tested it three ways:
- Detection went 100% → 98.6% → 97.8% across raw / your pass / pass+checker, against a design
that could not resolve anything under about four points. Genuine human prose was called human
143 times out of 143.
- An explicit "write more human" ruleset fooled 0 of 17 judges. Telling the model to simply
try harder fooled 0 of 15.
- Length changed nothing. A 2,940-word essay was caught 8/8 whole and 24/24 in six slices.
Then I took it to a commercial detector. Pangram 4.0 returned 100% AI on four variants,
including one where I changed zero words and only moved where sentences joined.
The finding related to your pattern list
Judge-cited tells, before and after your pass:
| pattern class |
before |
after |
| AI vocabulary |
70 |
0 |
| typography (dash / quotes) |
60 |
0 |
| uniform beat rate |
60 |
101 |
| aphoristic closers |
38 |
61 |
The pass crushes the lexical patterns on your list and grows the rhythmic ones. Ten named
tells in my data have no rule at all, and every one of them is rhythmic rather than lexical. My
read is that a subtractive rewrite makes prose more uniformly well-shaped, and uniform good shape
is itself a tell.
The repo description says the tool "removes signs of AI-generated writing." With 3,240 forks, I
suspect that some people may read that as defeats AI detection. My data says it does not, and that the
thing it actually (perhaps subjectively) does is makes the draft better to read.
Reproduce any of it yourself
Corpora, judgment files, tally scripts and the two numbers I had to correct mid-study:
https://github.com/itsryanlenk/humanist
Caveats: every judge is a language model from one family, which is the friendliest possible version of the detection test, in the future I'd like to run it more stringently. The quality study ran 77 of 144 planned pairwise trials after infrastructure failures. The human control is matched on register and length but not on subject.
I look forward to your input, positive or negative. I'm @itsryanlenk everywhere if it's easier.
Hello Mr. Chen,
First of all I want to thank you for your work on this plugin. Without it, I would not be writing this message to you now, nor would I have found some of the passions I am deeply obsessed with currently. I currently am building my own solo business in the realm of SEO, AEO and GEO which was born from a conversation I had with my mother about her brick and mortar business and her Shopify storefront. At some point during the journey I stumbled upon your humanizer plugin.
The culmination of that moment is what I'd like to convey to you here in this open issue. I ran a blind study with your tool in conjunction with my own version in great detail. If you have time, I'd love for you to dig into my project in detail. It was initially meant at some level to mask AI assisted prose, but eventually what was born from it is far more important I believe. I'll be including the details below. Again, thank you. I want you to know that this stranger out in the world wide web was truly inspired due to your work. The thing I found is that your pass is the only part of the whole study that had tangible evidence of a clear-cut win.
What worked
Blind pairwise, quality only, authorship never mentioned to the judges:
The third row is what I believe is something worth your time. My checker adds nothing measurable on top of your
pass. It emitted its input byte-identical in 11 of 18 cells, so twelve of those twenty-one
trials were judges comparing a file to itself.
Your rewrite pass carries the entire quantitative benefit. It is the sole component in the entire
study with an unambiguous positive result. I built the others on top of it.
What didn't work, which I believe matters for your README
I originally vendored this to try to beat AI detection. It does not, and I tested it three ways:
that could not resolve anything under about four points. Genuine human prose was called human
143 times out of 143.
try harder fooled 0 of 15.
Then I took it to a commercial detector. Pangram 4.0 returned 100% AI on four variants,
including one where I changed zero words and only moved where sentences joined.
The finding related to your pattern list
Judge-cited tells, before and after your pass:
The pass crushes the lexical patterns on your list and grows the rhythmic ones. Ten named
tells in my data have no rule at all, and every one of them is rhythmic rather than lexical. My
read is that a subtractive rewrite makes prose more uniformly well-shaped, and uniform good shape
is itself a tell.
The repo description says the tool "removes signs of AI-generated writing." With 3,240 forks, I
suspect that some people may read that as defeats AI detection. My data says it does not, and that the
thing it actually (perhaps subjectively) does is makes the draft better to read.
Reproduce any of it yourself
Corpora, judgment files, tally scripts and the two numbers I had to correct mid-study:
https://github.com/itsryanlenk/humanist
Caveats: every judge is a language model from one family, which is the friendliest possible version of the detection test, in the future I'd like to run it more stringently. The quality study ran 77 of 144 planned pairwise trials after infrastructure failures. The human control is matched on register and length but not on subject.
I look forward to your input, positive or negative. I'm @itsryanlenk everywhere if it's easier.