Skip to content

README.md

Latest commit

 

History

History
237 lines (172 loc) · 10.4 KB

README.md

File metadata and controls

237 lines (172 loc) · 10.4 KB
Raw

CodeQL Quality Gate Status

analyze-logs

Spring Boot web dashboard for Amazon CloudFront standard logs (JSON format). Fetches log files from S3, stores them in a local SQLite database, and displays interactive charts with drill-down detail pages.

Screenshot

Security notice

This application is intended for local, single-user use only.

  • There is no authentication or authorization. Anyone who can reach port 8080 can view the dashboard and trigger S3 fetches.
  • There is no CSRF protection. The application uses no Spring Security, so state-changing operations are not protected against cross-site request forgery. The Refresh from S3 action uses a GET request to mitigate this for the one write-like operation, but the application should not be exposed to untrusted networks.

Do not expose this application on a public interface or behind a shared reverse proxy without adding authentication (e.g. Spring Security with HTTP Basic, or an authenticating proxy such as nginx/Authelia).

Security controls in place

A security review found no exploitable vulnerabilities. The following controls are verified:

Area Control
SQL injection All database queries use JdbcTemplate with ? bind parameters. User-supplied values are never interpolated into SQL strings.
XSS (server-side) All Thymeleaf templates use th:text / th:content for user-controlled output, which applies automatic HTML entity escaping. th:utext is not used anywhere.
XSS (client-side) API responses are rendered onto <canvas> via Chart.js. Canvas drawing APIs do not interpret HTML or JavaScript. encodeURIComponent() is applied to all user-derived values placed into URLs.
Path traversal No file-serving endpoints with user-controlled paths exist. S3 object keys come from AWS API responses, not from user input.
Data exposure API endpoints return only CloudFront access-log data, which is the application's stated purpose. No credentials or internal state are exposed.

Build & run

Requires Java 25 and Maven.

mvn package
java -jar target/analyze-logs-1.0-SNAPSHOT.jar --spring.profiles.active=local

Or without packaging:

mvn spring-boot:run -Dspring-boot.run.profiles=local

Open http://localhost:8080.

Configuration

application.yml (committed — safe defaults)

Key properties:

Key Default Description
app.aws.region us-east-1 AWS region of the S3 bucket
app.aws.bucket `` S3 bucket containing CloudFront logs
app.aws.prefix `` S3 key prefix (e.g. AWSLogs/123456789/CloudFront/)
app.aws.profile `` AWS credentials profile (~/.aws/credentials); empty = default chain
app.db-path logs.db SQLite file path (relative to working directory)
server.port 8080 HTTP port
uri-stem-filter.excluded-extensions .css, .js, .png, … File extensions excluded from all URL charts (static assets)
referer-filter.self-referers [] Referer prefixes to exclude from Top Referers (your own domain). Matched with and without trailing slash, and as a bare domain without scheme.
referer-normalizer.rules Google, Bing, Yahoo, … Rules to group referer URLs into a single label. Each rule has label and one of domain (exact), domain-starts-with, or domain-ends-with.

application-local.yml (gitignored — your secrets)

Create src/main/resources/application-local.yml to override values without touching the committed file:

referer-filter:
  self-referers:
    - "https://your-site.example.com/"
    - "http://your-site.example.com/"

app:
  aws:
    bucket: "my-cloudfront-logs"
    prefix: "AWSLogs/123456789/CloudFront/"
    profile: "my-aws-profile"
  db-path: /absolute/path/to/logs.db

This file is listed in .gitignore and will never be committed. Activate it with --spring.profiles.active=local.

User-agent classification rules

The ua-classifier.rules list in application.yml maps UA substrings to human-readable labels. Rules are evaluated top-to-bottom; first match wins. Add entries to classify custom bots or internal tools:

ua-classifier:
  rules:
    - pattern: "MyInternalBot"
      label: "Internal crawler"

Dashboard

Bot Analysis page

Three detection methods, all scoped to the selected date range:

Section Detection signal
Top Probable Bots IPs that fetched /robots.txt and then crawled another URL within 1 hour using the same IP and user agent.
No Static Assets Bots IPs whose every request (per IP+UA pair) contains zero CSS, JS, WebP, ICO, or SVG fetches — real browsers always load static assets. Feed readers (/feed.xml, /rss.xml) are excluded.
Robots.txt Disobedient Bots User agents that requested disallowed paths after the current robots.txt rules were in place. Requires a Refresh Robots to load the latest rules.

The Bot vs Human Traffic Over Time chart shows daily request volume split between known bots (AI Bots, Search Bots, Other Bots, Apps) and human browsers.

Date range presets and custom date picker are shared with the main dashboard.

Main dashboard

Six charts, all scoped to the selected date range:

Chart Description
Top User Agents Horizontal stacked bar — classified UA names by request count, coloured by edge result type. Click a bar to open the UA detail page.
Top Blocked Countries (403) Horizontal bar — countries blocked with HTTP 403. Click a bar to open the country detail page.
Top Allowed URLs Horizontal bar — most-requested paths with status < 400; static assets (.css, .js, images, etc.) excluded.
Top Blocked URLs Horizontal bar — most-requested paths across all statuses; .php files grouped as PHP, /wp-* paths grouped as Wordpress.
Top Referers Horizontal bar — most frequent Referer headers; self-referrals and null referers excluded. Known search engines and social sites (Google, Bing, Yahoo, DuckDuckGo, Qwant, Facebook, Babelio) are grouped under a single label. Configurable via referer-normalizer.rules.
Requests per Day Stacked bar — daily breakdown by edge result type: Hit, Miss, Function, Redirect, Error.

Date range presets: Today / 7 days / 30 days / 3 months or a custom date picker.

Hide bots & apps toggle removes traffic from the AI Bots, Search Bots, Other Bots, and Apps UA groups — as well as entries with no user agent — from all six charts simultaneously. State is persisted in localStorage.

Refresh from S3 button triggers an incremental fetch (skips already-imported files).

UA detail page

Opened by clicking a bar in Top User Agents. Shows charts scoped to a single classified user-agent:

Chart Description
Result Types Pie — edge result type breakdown for this UA
Countries Pie — geographic distribution of requests
Top URLs Horizontal stacked bar — most-requested paths for this UA, coloured by edge result type (PHP / WordPress grouping applied)
Requests per Day Line — daily request trend by edge result type

Country detail page

Opened by clicking a bar in Top Blocked Countries. Shows charts scoped to a single country:

Chart Description
Result Types Pie — edge result type breakdown for this country
Top URLs Horizontal stacked bar — most-requested paths from this country, coloured by edge result type (PHP / WordPress grouping applied)
Requests per Day Line — daily request trend by edge result type

AWS credentials

No credentials are stored by this application. Authentication is delegated to the AWS SDK default credentials chain:

  1. Environment variables — AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
  2. ~/.aws/credentials — populated by aws configure
  3. EC2/ECS instance profile

Minimum required IAM permissions

{
  "Effect": "Allow",
  "Action": ["s3:ListBucket", "s3:GetObject"],
  "Resource": [
    "arn:aws:s3:::my-cloudfront-logs",
    "arn:aws:s3:::my-cloudfront-logs/*"
  ]
}

Findings

Investigation queries are in etc/queries/. Run them with:

sqlite3 logs.db < etc/queries/suspicious_user_agents.sql

Suspicious user agents (PHP / WordPress scanners)

Agents hitting .php or /wp* URLs 10+ times per day, with no legitimate traffic on the site:

User Agent Date Hits
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/71.0.3542.0 Safari/537.36 2026-04-07 15

Requests with an empty user_agent are excluded upfront — they are always suspicious by definition.

Database

Logs are stored in a SQLite file (logs.db by default, relative to the working directory). Query it directly with any SQLite-compatible tool:

sqlite3 logs.db "SELECT ua_name, COUNT(*) FROM cloudfront_logs GROUP BY ua_name ORDER BY 2 DESC"

Schema

cloudfront_logs (
    id                        INTEGER PRIMARY KEY,
    timestamp                 TEXT NOT NULL,   -- ISO-8601 UTC
    edge_location             TEXT,
    sc_bytes                  INTEGER,
    client_ip                 TEXT,
    method                    TEXT,
    uri_stem                  TEXT,
    status                    INTEGER,
    referer                   TEXT,
    user_agent                TEXT,
    edge_result_type          TEXT,
    protocol                  TEXT,
    cs_bytes                  INTEGER,
    time_taken                REAL,
    edge_response_result_type TEXT,
    protocol_version          TEXT,
    time_to_first_byte        REAL,
    edge_detailed_result_type TEXT,
    content_type              TEXT,
    content_length            INTEGER,
    country                   TEXT,            -- ISO 3166-1 alpha-2
    ua_name                   TEXT             -- classified user-agent label
)

fetched_files (
    s3_key     TEXT PRIMARY KEY,
    fetched_at TEXT
)