Skip to content

HTTP_RESPONSE event data is built by two divergent functions #3386

Description

@liquidsec

Two functions build the dict for an HTTP_RESPONSE event, and they disagree:

  • response_to_event_dict() in bbot/core/helpers/web/response_event.py, used by http and lightfuzz
  • WebHelper.response_to_json() in bbot/core/helpers/web/web.py, used by wayback

They're near-duplicates but not duplicates, so the same response yields different event
data depending on which module found it.

Key sets differ:

  • only in response_to_event_dict: input, content_length, title, location, cert_info
  • only in response_to_json: timestamp, port, scheme

raw_header is a different shape. For one response:

response_to_event_dict: 'HTTP/1.1 200 \r\nContent-Type: text/html\r\nServer: nginx\r\n\r\n'
response_to_json:       'Content-Type: text/html\r\nServer: nginx'

HTTP_RESPONSE.raw_response is raw_header + body, so on the response_to_json path it
has no status line and no blank line between headers and body.

Duplicate header names. response_to_event_dict joins them, response_to_json keeps
the last. For two Set-Cookie headers:

response_to_event_dict: 'a=1, b=2'
response_to_json:       'b=2'

response_to_json recomputes hashes blasthttp already has. It runs md5/mmh3/sha256
over the body and headers itself; response.hash is the same values, computed lazily in
Rust and cached. (The values do match, I checked. It's wasted work, not a discrepancy.)

Smaller ones: body is response.body vs smart_decode(response.content); content_type
is the raw header value vs the value split on ; and stripped; header keys are
k.lower().replace("-", "_") vs tagify(k, delimiter="_").

Net effect: wayback's HTTP_RESPONSE events are missing title, location,
content_length and cert_info, and their raw_response is malformed.

Noticed while adding decode_error in #3385, which had to be written into both.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions