Skip to content

feat: [generate-openfilters-scanners.py] generate scanner warninglists from OpenFilters/internet-scanners - #389

Merged
adulau merged 2 commits into
MISP:mainfrom
elhoim:warninglists/openfilters-scanners-generator
Aug 30, 2026
Merged

feat: [generate-openfilters-scanners.py] generate scanner warninglists from OpenFilters/internet-scanners#389
adulau merged 2 commits into
MISP:mainfrom
elhoim:warninglists/openfilters-scanners-generator

Conversation

@elhoim

@elhoim elhoim commented Aug 29, 2026

Copy link
Copy Markdown
Member

Adds a generator that builds the internet-wide-scanner warninglists from
OpenFilters/internet-scanners,
and wires it into generate_all.sh.

That repository tracks the ranges scanning organisations themselves publish as
their scan origins, and ships them machine-readable under cidr/<org>_v4.txt
and cidr/<org>_v6.txt. It is released into the public domain (Unlicense) and
is actively maintained (last upstream commit 2026-08-20).

Several MISP scanner lists are currently maintained entirely by hand, and 56
organisations tracked upstream have no published-range list here at all. This
gives all of them a single refresh path.

Design notes

One generator, many lists. Follows the precedent of generate-microsoft-azure.py
and generate-vpn.py. The upstream repository is a single snapshot, so one
download serves every list and they stay mutually consistent.

Fetches the repository tarball, not 130 individual raw files: one request,
an atomic snapshot, and no GitHub raw rate-limiting.

Organisations are mapped explicitly in ORG_TO_LIST rather than deriving a
directory name from the upstream filename. When upstream adds an organisation
the generator logs a warning naming it and skips it, instead of silently
creating a list nobody named or reviewed.

Existing lists are only ever added to. Entries already committed are merged
in before consolidate_networks() runs. Several of these lists were curated by
hand from sources this one does not cover, and a range an organisation scanned
from in the past is still worth recognising in a false-positive warninglist.
Verified: no existing entry is dropped from any of the six lists this touches.

Pre-existing lists keep their name and description. MISP keys
warninglists on their name, so renaming one in place would appear server-side as
a second, competing list rather than an update. Only list and version change
for lists that already exist.

Malformed ranges are dropped, not widened. Upstream currently ships one
entry with host bits set (34.19.127.176/27, under mandiantasm). Accepting it
would mean widening it to its enclosing /27, which in a warninglist means
suppressing alerts for 16 addresses the operator never claimed. The generator
rejects it with a logged warning instead; the warning is there so it can be
reported upstream.

Not taken from this source

Organisation Reason
driftnet, modat, onyphe already have their own generators
other unattributed grab-bag; upstream moves entries out of it as they get attributed, which would churn two lists per attribution
inactive/ (incl. netsecscan) upstream marks these retired — netsecscan-scanning therefore stays hand-maintained

Also note this data has a different provenance from the *-nt-scanning
lists, which come from CIRCL network-telescope observations. Published ranges
and observed ranges legitimately differ, so the two families are kept separate
and this generator never writes to a -nt-scanning list.

The first generation of the new lists, and the updates to the six existing ones,
are submitted as separate pull requests.

Verified: ./validate_all.sh passes, and the generator exits 0 with a clean
generators.log apart from the intended malformed-range warning.

@adulau
adulau merged commit db2400f into MISP:main Aug 30, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants