Two functions build the dict for an HTTP_RESPONSE event, and they disagree:
response_to_event_dict() in bbot/core/helpers/web/response_event.py, used by http and lightfuzz
WebHelper.response_to_json() in bbot/core/helpers/web/web.py, used by wayback
They're near-duplicates but not duplicates, so the same response yields different event
data depending on which module found it.
Key sets differ:
- only in
response_to_event_dict: input, content_length, title, location, cert_info
- only in
response_to_json: timestamp, port, scheme
raw_header is a different shape. For one response:
response_to_event_dict: 'HTTP/1.1 200 \r\nContent-Type: text/html\r\nServer: nginx\r\n\r\n'
response_to_json: 'Content-Type: text/html\r\nServer: nginx'
HTTP_RESPONSE.raw_response is raw_header + body, so on the response_to_json path it
has no status line and no blank line between headers and body.
Duplicate header names. response_to_event_dict joins them, response_to_json keeps
the last. For two Set-Cookie headers:
response_to_event_dict: 'a=1, b=2'
response_to_json: 'b=2'
response_to_json recomputes hashes blasthttp already has. It runs md5/mmh3/sha256
over the body and headers itself; response.hash is the same values, computed lazily in
Rust and cached. (The values do match, I checked. It's wasted work, not a discrepancy.)
Smaller ones: body is response.body vs smart_decode(response.content); content_type
is the raw header value vs the value split on ; and stripped; header keys are
k.lower().replace("-", "_") vs tagify(k, delimiter="_").
Net effect: wayback's HTTP_RESPONSE events are missing title, location,
content_length and cert_info, and their raw_response is malformed.
Noticed while adding decode_error in #3385, which had to be written into both.
Two functions build the dict for an HTTP_RESPONSE event, and they disagree:
response_to_event_dict()inbbot/core/helpers/web/response_event.py, used byhttpandlightfuzzWebHelper.response_to_json()inbbot/core/helpers/web/web.py, used bywaybackThey're near-duplicates but not duplicates, so the same response yields different event
data depending on which module found it.
Key sets differ:
response_to_event_dict:input,content_length,title,location,cert_inforesponse_to_json:timestamp,port,schemeraw_headeris a different shape. For one response:HTTP_RESPONSE.raw_responseisraw_header + body, so on theresponse_to_jsonpath ithas no status line and no blank line between headers and body.
Duplicate header names.
response_to_event_dictjoins them,response_to_jsonkeepsthe last. For two
Set-Cookieheaders:response_to_jsonrecomputes hashes blasthttp already has. It runs md5/mmh3/sha256over the body and headers itself;
response.hashis the same values, computed lazily inRust and cached. (The values do match, I checked. It's wasted work, not a discrepancy.)
Smaller ones:
bodyisresponse.bodyvssmart_decode(response.content);content_typeis the raw header value vs the value split on
;and stripped; header keys arek.lower().replace("-", "_")vstagify(k, delimiter="_").Net effect:
wayback's HTTP_RESPONSE events are missingtitle,location,content_lengthandcert_info, and theirraw_responseis malformed.Noticed while adding
decode_errorin #3385, which had to be written into both.