Skip to content

pHash.md

Latest commit

 

History

History
127 lines (81 loc) · 6.57 KB

pHash.md

File metadata and controls

127 lines (81 loc) · 6.57 KB
Raw

TLDR - just some AI dump my notes to a file, but pHashing only worked well on Favicons based on letters and not well on others.

pHash Background

A pHash, or perceptual hash, is a digital fingerprint based on how an image looks rather than on the exact bytes of the file. Cryptographic hashes such as MD5 and SHA-256 change completely if one byte changes. A pHash is different: resized, recompressed, or slightly altered images can still produce similar pHashes.

Image pHash generation usually follows this broad shape:

  1. Convert to grayscale so the hash focuses on light, shadow, and structure rather than color noise.
  2. Resize to a small fixed size, commonly around 32x32, to remove fine pixel-level detail.
  3. Apply a Discrete Cosine Transform, or DCT, to move from raw pixels into frequency information.
  4. Keep the low-frequency area, often the top-left 8x8 block, because it represents the broad visual structure.
  5. Compare those 64 values against their average to create a 64-bit hash.

Common implementations differ in small but important ways. Some compare the low-frequency DCT values against an average; others compare against a median. Some exclude the first DCT value, also called the DC coefficient, because it represents overall brightness. These choices can produce different pHash strings for the same image, so pHash values should be compared only when they were generated with the same implementation and settings.

To compare two pHashes, use Hamming distance: count how many bits differ between the two 64-bit values. A distance of 0 means the pHashes are identical. Low distances often indicate visually similar favicons; high distances usually mean the images are unrelated.

In local experiments, a max distance around 14 was useful for broader hunting. That is looser than the UI default and may return noisy results, but it can be helpful when looking for related favicon families instead of near-identical icons.

pHash Generation

pHash values can be generated with scripts/hash_images.py using Pillow and ImageHash:

phash = str(imagehash.phash(img))

The result is a 16-character hexadecimal string. It represents a 64-bit perceptual hash, so visually similar favicons should have pHashes with a small Hamming distance.

The exact conventional behavior used here is:

  • Default hash_size=8, so the final hash is an 8x8 / 64-bit value.
  • Default highfreq_factor=4, so the image is resized to 32x32 before DCT.
  • The image is converted to grayscale with image.convert("L").
  • The image is resized to 32x32 using LANCZOS resampling.
  • The code computes a 2D DCT and keeps the top-left 8x8 low-frequency block.
  • The median is calculated from the low-frequency block (including the DC coefficient).
  • All 64 low-frequency values are compared against that median to create the bit array.
  • The stored string is ImageHash's 16-character lowercase hex representation of that bit array.

Stored pHash Algorithms

Both algorithms use identical DCT-based recipes (32×32 resize, 8×8 low-frequency block). The only substantive difference is the threshold:

Previous mean/DC-excluded (phash_legacy) ImageHash-compatible median (phash)
Threshold numpy.mean(dct_low_freq[1:]) numpy.median(dct_low_freq)
DC coefficient Excluded from threshold Included in median
Matches find_phash_matches(..., algorithm="legacy") find_phash_matches(..., algorithm="current")

I keep the values algorithm=current and algorithm=legacy for compatibility with existing links. In the UI and docs, current means ImageHash-compatible median pHash, while legacy means the previous mean/DC-excluded pHash.

Practical note: for ImageHash-compatible median algorithm, use max_distance=6–8 as a starting point. For the previous mean/DC-excluded algorithm, max_distance=10–12 can compensate for the threshold difference while maintaining fewer false positives than very broad searches.

Practical compatibility note: if another tool uses median instead of mean, includes the DC coefficient in the mean, uses a different resize filter, or hashes a different decoded frame from an ICO file, its pHash may not match this database exactly.

Searching by pHash Distance

I had a UI to search Hamming distance:

distance = (int(target_phash, 16) ^ int(candidate_phash, 16)).bit_count()

Suggested values:

N Meaning
0 Exact pHash match
1-4 Very close visual matches
5-8 Good exploratory range
9-12 Loose visual matches
13-14 Broad hunting range that may still be useful in experiments
>14 Often noisy; use for very broad exploration

Recursive pHash Expansion

Direct pHash Search answers: "what looks like this seed favicon?" It compares every candidate to the original seed only.

Recursive pHash Expansion answers: "what favicon cluster does this seed belong to?" It treats each match as a new search seed. If favicon A is within N bits of favicon B, and B is within N bits of favicon C, then C is included even when C is farther than N bits from A.

Conceptually, this builds a similarity graph and I just wanted to see how legit it could be:

  • Each favicon pHash is a node.
  • An edge exists when two pHashes have Hamming distance less than or equal to N.
  • Recursive expansion walks that graph from a seed favicon.
  • Cluster expansion keeps walking until no new reachable neighbors remain or the API limit is hit.

Expansion results include both the parent-hop distance and the original seed distance:

Field Meaning
seed_mmh3 / seed_phash Original favicon used to start the walk
matched_mmh3 / matched_phash Favicon included in the expanded result set
parent_mmh3 / parent_phash Previous favicon that discovered this match
distance_from_parent Hamming distance for the graph edge; always <= distance
distance_from_seed Hamming distance from the original seed; may be larger than distance
depth Number of recursive hops from the seed
algorithm current for median pHash, legacy for previous mean/DC-excluded pHash

Shodan mmh3 Hashes

The mmh3 value is the same signed integer used by Shodan's http.favicon.hash search and is already stored as the favicon hash in this database.

Example recomputation shape:

import base64
import re
import mmh3

with open("favicon.ico", "rb") as favicon:
    b64 = base64.b64encode(favicon.read()).decode("utf-8")
    wrapped = re.sub("(.{76}|$)", "\\1\n", b64, 0, re.DOTALL)
    print(mmh3.hash(wrapped))