loglayzr

Loglayzr — flag the noise, find the attacks.

A zero-dependency Python toolkit for scanning Apache/Nginx combined-format access logs for suspicious activity. Uses signature-based pattern matching, GeoIP enrichment, and interactive browsing to surface reconnaissance, exploitation attempts, and brute-force attacks hiding in your logs.

Features

• 57 built-in detection patterns covering SQL injection, command injection, path traversal, web shells, CVE probes, WordPress attacks, credential stuffing, config scraping, XSS, scanner user-agents, and more
• GeoIP enrichment — tags matches with country codes; whitelist your own countries to suppress noise
• IP whitelist — ignore known-safe IPs (internal servers, monitoring)
• Static asset noise filter — skips legitimate CSS/JS/image loads from real page views
• Directory mode — pass a directory instead of a file to scan all logs recursively (handles .gz transparently)
• Two interfaces — summary report (analyze.py) and interactive browser (browse.py)
• Pipeline-friendly — JSONL output, pipe to jq, grep, or any Unix tool
• All configuration in one file — log format, whitelists, GeoIP command, filter settings live in config.json

Requirements

• Python 3.12+
• geoiplookup from the geoip-bin package (for GeoIP lookups)

sudo apt install geoip-bin geoip-database

No Python packages outside the standard library.

Quick start

# Summary of all suspicious activity in one file
python3 analyze.py logs/access.log

# Or process an entire directory (recursive, .gz supported)
python3 analyze.py logs/

# Save to a report file
python3 analyze.py logs/access.log > reports/site.report.txt

# Only critical hits
python3 analyze.py logs/ | grep '"sev":"CRITICAL"'

# Browse matches interactively (select category → pattern → browse)
python3 browse.py logs/access.log

# Aggregate by country across all logs in a directory
python3 analyze.py logs/ | grep '^{' | \
    jq -s 'group_by(.cc) | map({cc: .[0].cc, count: length}) | sort_by(-.count)'

analyze.py — summary report

python3 analyze.py <file|directory>

Accepts a single log file or a directory (walked recursively, .gz files decompressed on the fly, hidden files and known binary extensions skipped). All matches are aggregated into a single combined summary.

Single-file summary:

=== SUMMARY: access.log ===

Multi-file summary:

=== SUMMARY: logs/ (42 files) ===

Full example:

=== SUMMARY: access.log ===

Lines parsed:    9,277
Suspicious hits: 1,613

  By severity:
    CRITICAL:  80
        HIGH:  388
      MEDIUM:  230
         LOW:  915

  By category (top 10):
    config_scraping      534
    wordpress            407
    ...

  Top attacking IPs:
    136.109.155.86     58 [US]  (env_file_probe×28, ...)
    213.209.159.175    58 [US]  (env_file_probe×58)

  By country (non-whitelisted):
    US     1234
    CN      456
    ...

browse.py — interactive investigation

python3 browse.py <file|directory>

Also accepts a file or directory. Parses all logs, then presents menus to choose a category and pattern, then lets you browse matches one at a time.

Key	Action
N / Enter / Space	Next match
P / Backspace	Previous match
B	Back to category/pattern menu
Q / Ctrl+C	Quit (works from menus too)

Each match shows severity (color-coded), pattern description, full log entry, and country code for non-whitelisted IPs.

Patterns

Detection rules live in patterns.json. Each pattern has:

Field	Description
name	Unique identifier (e.g. env_file_probe)
category	Grouping label (config_scraping, sqli, rce, …)
severity	CRITICAL, HIGH, MEDIUM, or LOW
regex	Python regex applied to the target field (or null for length checks)
field	Which log field to match: url, user_agent, status, method, referrer
type	"length_check" (optional) — instead of regex, flags entries where the field exceeds threshold characters
description	Human-readable explanation shown in browse mode

57 patterns across 13 categories: config_scraping (10), recon (8), wordpress (6), cve_exploit (6), sqli (5), rce (5), anomaly (4), xss (3), admin_panel (3), evasion (3), path_traversal (2), backdoor (1), php_exploit (1).

Add new patterns by editing patterns.json — no code changes needed.

Configuration

All configurable settings live in config.json:

{
    "log_format": {
        "regex": "^(\\S+) \\S+ \\S+ \\[([^\\]]+)\\] \"(\\S+) (\\S+) (\\S+)\" (\\d{3}) (\\S+) \"([^\"]*)\" \"([^\"]*)\"",
        "timestamp_format": "%d/%b/%Y:%H:%M:%S %z"
    },

    "patterns_file": "patterns.json",

    "whitelists": {
        "countries": ["CN", "IN", "LK"],
        "ips": ["192.168.1.2"]
    },

    "geoip": {
        "command": "geoiplookup",
        "regex_search": "GeoIP Country Edition:\\s*([A-Z]{2})"
    },

    "filters": {
        "static_extensions": [
            ".js", ".css", ".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp",
            ".ico", ".woff", ".woff2", ".ttf", ".eot", ".mp4", ".mp3", ".webm",
            ".pdf", ".zip", ".gz", ".tar"
        ]
    }
}

Section	Setting	Default	Description
log_format	regex	(combined log format)	Python regex to parse log lines. Must have 9 capture groups for IP, timestamp, method, URL, protocol, status, size, referrer, user-agent.
log_format	timestamp_format	%d/%b/%Y:%H:%M:%S %z	strptime format for the timestamp in log entries.
patterns_file	—	patterns.json	Path to the pattern definitions file. Relative paths resolve from the config file's directory.
whitelists	countries	["CN", "IN", "LK"]	Two-letter country codes to ignore. Matches from these IPs are silently dropped.
whitelists	ips	["192.168.1.2"]	Specific IP addresses to ignore (e.g. internal hosts, monitoring services).
geoip	command	geoiplookup	Command to run for GeoIP lookups. Swap to mmdblookup, geoiplookup6, or any tool that outputs the country code on stdout.
geoip	regex_search	GeoIP Country Edition:\s*([A-Z]{2})	Regex to extract the two-letter country code from the GeoIP command's output. First capture group must yield the code.
filters	static_extensions	(see above)	File extensions that trigger the static-load noise filter when combined with a GET method and non-empty referrer.

Missing or misconfigured settings fall back to sensible defaults — config.json can be sparse.

Filter pipeline

Each log line passes through filters in order before pattern matching:

1. Log file resolution — if a directory was passed, walk it recursively. Skip hidden files/dirs and known binary extensions. Decompress .gz on the fly.
2. Static load — GET + static extension (.js, .css, .png, …) + non-empty referrer → skip
3. IP whitelist — IP in IP_WHITELIST → skip
4. Country whitelist — IP resolves to a country in COUNTRY_WHITELIST → skip
5. Pattern matching — remaining entries checked against all 57 patterns

Filters are defined in common.py — add new ones in should_skip().

File structure

loglayzr/
├── analyze.py        # Summary report script
├── browse.py         # Interactive match browser
├── common.py         # Shared filtering, match construction, file resolution
├── config.py         # Configuration loader (reads config.json)
├── config.json       # User-facing configuration file
├── geoip.py          # GeoIP lookups with caching
├── patterns.json     # Suspicious pattern definitions (57 rules)
├── LICENSE           # GPLv3
└── README.md

License

GNU General Public License v3.0 — see LICENSE.