When Soup is passed a correctly decoded WideString, like customer’s (where the apostrophe is character 8217), it tries to decode it, which turns it into mojibake: customerâ��s.
It seems the problem comes from the port from Python's Beautiful Soup, which accepts raw bytes instead of strings and uses "Unicode, Dammit" to correctly decode them.
Since this Soup variant only accepts strings, it seems we should just eliminate the decoding step (i.e. remove self decode from SoupParser>>#parseString:. I commented it out and all the tests still pass. Not an expert, but it seems decoding cannot work in any general way on already-decoded strings.
Longer term, it could also be interesting to extend Soup with a layer accepting bytes and attempting to decode them, but that seems outside the scope of this particular issue.
cc/ @Ducasse @svenvc (resident decoding expert)
When Soup is passed a correctly decoded WideString, like
customer’s(where the apostrophe is character 8217), it tries to decode it, which turns it into mojibake:customerâ��s.It seems the problem comes from the port from Python's Beautiful Soup, which accepts raw bytes instead of strings and uses "Unicode, Dammit" to correctly decode them.
Since this Soup variant only accepts strings, it seems we should just eliminate the decoding step (i.e. remove
self decodefromSoupParser>>#parseString:. I commented it out and all the tests still pass. Not an expert, but it seems decoding cannot work in any general way on already-decoded strings.Longer term, it could also be interesting to extend Soup with a layer accepting bytes and attempting to decode them, but that seems outside the scope of this particular issue.
cc/ @Ducasse @svenvc (resident decoding expert)