Not detecting word correctly?

[ktbarrett]

I have the word pybind11 in my text and have added pybind11 to my spelling list. However, it seems to ignore the trailing numbers and think the word is pybind only, creating a spelling error. I could add pybind to the spelling list, but then pybind12 would be valid, which isn't ideal. Why aren't the trailing numbers considered to be a part of the word? I didn't see any configuration option about changing that behavior.

[dhellmann]

The pyenchant tokenizer does not consider digits as being part of the character set making up valid words, so they are removed from the tokens it returns.

There is an option to pass additional valid characters to the tokenizer, but that is not exposed by the sphinx integration layer. I don't think it would help in your case, because the word cannot start or end with those extra characters (the feature is meant to handle situations like apostrophes).

You could write a new tokenizer that did include digits in the set of characters making up valid words, but that might be extreme for your case. I think for that to work properly you would need to have the tokenizer accepted in pyenchant so that its name can be used with get_tokenizer().

It looks like pybind11 is the name of a package. The PyPI package name filter won't work because the trailing 11 are removed so the package name won't match the token. What I usually do in cases like that is either add the base package name to the spelling list or add literal markup around the name in the text to make the spell checker ignore it.

[dmerejkowsky]

Hi, PyEnchant maintainer here.

You could write a new tokenizer that did include digits in the set of characters making up valid words, but that might be extreme for your case

I concur. Also, writing a tokenizer is rarely fun.

What I usually do in cases like that is either add the base package name to the spelling list or add literal markup around the name in the text to make the spell checker ignore it.

I think this is the way to go :)

[dhellmann]

Thanks for the advice, @dmerejkowsky, and for maintaining PyEnchant!

[webknjaz]

I just want to add that when it sees words like isn't, sphinxcontrib.spellcheck reports isn as a mistake.
Also, I've hit a rather interesting case of dynamically generated identifiers in titles. In particular, I have a changelog document that renders a dynamically generated section title with 0.2.7.dev72+gb0796fb in it. This project then complains about dev and gb being invalid. I can ignore dev/alpha/beta to fix the first part but it's more complicated with the SCM identifier. The first letter g means it's a Git repository and the following characters represent an abbreviated commit SHA. So there's a lot of combinations with g+[1-9a-f] possible here and listing this many is not really sustainable.
@dhellmann do you think you could expose something to deal with it? Best if it could be specified per document or even more precise. I cannot even use a custom spelling_filters implementation when the tokenizer has a hard time recognizing the full version as a single word. Do you think you could fire an event maybe that would get a callback to modify the string before you pass it to pynchant? Maybe I'd just cut the word out in that callback and the rest would be linted.

P.S. Here's MyST Markdown code snippet that I have:

## [{sub-ref}`release`] - as of {sub-ref}`today` _{subscript}`/UNRELEASED DRAFT/`_

[webknjaz]

Just realized that the misspelled sequences may also be in the middle of the commit hash: 0.2.7.dev72+g3cdb8a9 has cdb reported as a mistake.

[ktbarrett]

The pyenchant tokenizer is basic in what it's capable of, but involves a lot of scaffolding code. Trying to further leverage that code is not a good way forward. A proper tokenizer solution would take advantage of user dictionaries to extend tokenization. Something like Aho-Corasick, and if it fails, backtracking and falling back on the current solution of language-specific lexeme matching and passing to enchant.

EDIT: Depending on the performance, startwith may suffice.
EDIT 2: startswith may have problems, it would have to check the word match is followed by a language-specific lexeme terminal character (space, punctuation, dashes, not numbers per the original issue, etc.) So I guess that means this functionality should be incorporated into pyenchant.

[tgross35]

What would need to happen and (and in which repo) to get this to work? I'll happily code something if it means that "doesn't" and "ad-hoc" officially become words again

[tgross35]

Looks like maybe the blocker is here pyenchant/pyenchant#286

@dmerejkowsky any chance you'd be able to take a look at that issue?

[dhellmann]

See #147 also, but yes, we need a supportable API to set the word separators.

[tgross35]

Thanks for being so responsive. Too bad that it seems like pyenchant is kind of in coast mode now