Example and workaround shown here:
# download file from a web server that does not explicity specify character encoding within content-type header
raw = get_url("http://1.2.3.4/source.txt")
# raw might contain special unicode symbols...
response = raw.decode("utf-8") # ...so decode it stritcly as utf-8
# response should now contain the proper characters as intendedBackground and resolution
In the rare situation that a web response does not explicity include the character encoding alongside the MIME type, get_url falls back to ISO-8859-1 decoding (*). That involves treating every byte as one character which can lead to mangled strings, e.g.
"// ï‚Ä¢·¥•‚Ä¢ î" instead of "// ʕ•ᴥ•ʔ".
(*) If the MIME type is JSON or XML related it does currently get treated as UTF-8.
In the source that processing is done here:
|
// try using the given encoding or a straight 8-bit widening (for convenience ISO-8859-1 does that trick) |
In hindsight it's probably more useful and safer if it simply defaults to UTF-8 encoding instead since it's ubiquitous for almost all text-based content these days.
It's safer since it might not be obvious at first whether any extended characters may trip up any parsing done.
This change in behaviour should be backwards compatible with existing recipes however some that make extensive use of get_url should be checked especially if it's not JSON or XML that they're consuming. Perhaps this one Microsoft Exchange schedule retriever.
To workaround this issue, you can simple decode the result that's returned. For example:
Example and workaround shown here:
Background and resolution
In the rare situation that a web response does not explicity include the character encoding alongside the MIME type,
get_urlfalls back to ISO-8859-1 decoding (*). That involves treating every byte as one character which can lead to mangled strings, e.g."// ï‚Ä¢·¥•‚Ä¢ î"instead of"// ʕ•ᴥ•ʔ".(*) If the MIME type is JSON or XML related it does currently get treated as UTF-8.
In the source that processing is done here:
nodel/nodel-jyhost/src/main/java/org/nodel/http/impl/ApacheNodelHttpClient.java
Line 308 in 415a00e
In hindsight it's probably more useful and safer if it simply defaults to UTF-8 encoding instead since it's ubiquitous for almost all text-based content these days.
It's safer since it might not be obvious at first whether any extended characters may trip up any parsing done.
This change in behaviour should be backwards compatible with existing recipes however some that make extensive use of
get_urlshould be checked especially if it's not JSON or XML that they're consuming. Perhaps this one Microsoft Exchange schedule retriever.To workaround this issue, you can simple decode the result that's returned. For example: