Skip to content

AGENTS.md

Latest commit

 

History

History
175 lines (115 loc) · 12.1 KB

AGENTS.md

File metadata and controls

175 lines (115 loc) · 12.1 KB
Raw

AGENTS.md

This file provides guidance to AI agents when working with code in this repository.

Project Overview

This is a simple static website hosted on GitHub Pages for the domain fuckfortyseven.org. The site serves as a placeholder/landing page with political content.

Architecture

  • Static HTML Site: Single-page website using plain HTML, CSS, and SVG
  • GitHub Pages Deployment: Uses app/ directory for GitHub Pages hosting, deployed via .github/workflows/static.yml
  • Domain: Custom domain fuckfortyseven.org configured via repository Pages settings

File Structure

  • app/index.html: Main HTML page with inline CSS and SVG graphics
  • app/img/trump.png: Image asset referenced in the HTML
  • README.md: Basic project title
  • LICENSE: Public domain (Unlicense)

Development

This project handles dependencies and package management via uv.

The site will be hosted on GitHub Pages.

Only htmx and alpine.js are used for frontend interactivity; it will be a single-page static site.

All content is generated by the main.py script.

Articles are stored in a persistent DuckDB database (utils/db.py), keyed by url, and exported to Parquet (articles.parquet / filtered_articles.parquet) as the interchange format for downstream consumers.

See docs/ai.md for the LLM sentiment judge that gates which DJT-related articles are considered negative enough to display.

utils/retry.py provides http_request_with_retry and feed_with_retry — use these instead of bare requests.get or feedparser.parse calls so transient failures (429, 5xx, ConnectionError) are retried with exponential backoff before a source is skipped.

utils/ratelimit.py provides RateLimiter, a sliding-window limiter — utils/newsapi.py and utils/rss.py each hold a module-level instance and call .acquire() before every outbound request so the pipeline stays under quota independent of the retry/backoff behavior above. Configure via RATE_LIMIT_REQUESTS (default 100) and RATE_LIMIT_INTERVAL_SEC (default 3600) env vars.

Each run also archives a timestamped Parquet snapshot of both stores under archive/<store>/YYYY-MM-DD/YYYY-MM-DDTHHMM.parquet (ArticleDB.archive_snapshot, wired in main.py) — a persistent, git-committed history of what was fetched/published over time, since the live .duckdb/.parquet files are ephemeral and gitignored. Query the whole archive with read_parquet('archive/**/*.parquet'). utils/render.py (run via uv run python -m utils.render) does two things: render_index injects the newest filtered_articles.duckdb rows into app/index.html in place, between the <!-- ARTICLES:BEGIN/END --> markers (a JSON payload for the Alpine poster rotation plus a noscript link list; if the store is missing the page is left untouched), and render_archive turns the filtered_articles archive into a static htmx-driven history view at app/archive/. See backlog task-011 for the design rationale.

The frontend (app/index.html) renders each article as a full-viewport pop-art poster per PRODUCT.md/DESIGN.md: a duotone-treated DJT portrait (SVG #silkscreen filter + CSS halftone overlay), Anton headline paste-up strips, Special Elite body text, hard-cut rotation via Alpine. Fonts are self-hosted woff2 in app/fonts/; portrait sources and their fair-use attribution live in app/img/djt/ (attribution.json).

utils/scrape.py provides fetch_html (curl_cffi with Chrome TLS impersonation, optional proxy) and extract_metadata (trafilatura, returns title/description/published_at) for uv run ./cli.py add-article <url>, which manually ingests one article into the DuckDB store — useful for paywalled/obscure sources the automated pipeline misses. Any field extraction misses is prompted for interactively (or --title/--description/--date/--source to supply it directly); if stdin isn't a TTY and fields are still missing, the command exits non-zero listing them. Duplicates are reported via the existing ArticleDB dedup-by-url path, no error.

utils/upnote.py reads a local UpNote install's SQLite store read-only (copy_db_readonly copies the .sqlite3 + -wal/-shm sidecars before ever opening anything, so a running UpNote instance is never touched) to power uv run ./cli.py import-upnote --db <staging.duckdb> — a one-time backfill of notes tagged #fuck45 or mentioning "trump" into a staging DuckDB (never the live store directly) for review before merging. UpNote's x-callback-url scheme is write/navigate-only and cannot return note content, so this is the only programmatic read path. extract_url is deliberately high-precision only: it trusts only a standalone source-URL line near the top of a note's text (the web clipper's own link line) and never falls back to "any href in the note's html", since most matching notes are full article clips with many unrelated in-body/author-bio/footnote links that would otherwise get mistaken for the source (see backlog task-014). is_excluded_source filters out video/Q&A platforms (YouTube, Quora) — the backfill is scoped to news articles only. Notes without a confident URL, and excluded-domain URLs, are recorded via eliot structured logging (import_upnote_unresolved/import_upnote_excluded/import_upnote_result message types nested under an import_upnote action, routed to --log — default import_upnote.log, gitignored) rather than flat report files; --dry-run previews the candidate/unresolved/excluded split before any fetching.

uv run ./cli.py merge-reviewed --staging-db <staging.duckdb> is the direct path for content that's already been human-reviewed (e.g. after eyeballing/pruning an import-upnote staging DB): it inserts straight into both articles.duckdb and filtered_articles.duckdb, bypassing main.py's automated DJTNewsFilter and sentiment-judge gate entirely, then calls utils/render.py:render_index() to refresh app/index.html, and by default commits and pushes that file (--no-push to stop after the local render and push yourself). It also marks each merged row manual_review = TRUE via ArticleDB.mark_manual_review() (a boolean column on articles, migrated into pre-existing stores automatically) — main.py's pipeline fully rebuilds filtered_articles.duckdb from scratch on every run (clear_all_articles + reinsert only what currently passes the DJT filter and negative-sentiment gate), so without this pinning a manually-approved row would only survive until the next 8-hourly cron run silently dropped it again. main.py unions in every manual_review = TRUE row from articles.duckdb when rebuilding the filtered store, regardless of that run's filter/sentiment outcome.

Content curation

The site theme is "All bad news, all the time. Negative DJT coverage, silkscreened daily." An article belongs only if it is (a) about Trump / his administration and (b) negative or damaging in framing. Wire filler with no Trump connection (obituaries, science, foreign economics, positive market news) must be removed even if it slips through the keyword/sentiment filters.

Permanent removal — uv run ./cli.py -r <value> [<value> ...] (--force skips the confirmation prompt; --no-push skips the automatic commit+push so you can craft the commit yourself). Each value can be:

  • A full source URL (trailing slash is normalized before comparison)
  • A hyphenated slug as rendered in the poster (e.g. trump-administration-rolls-back-dozens-of-gun-regulations) — matched bidirectionally against the computed title slug
  • Any bare substring matched case-insensitively against url, title, or source

The command appends to exclude_urls.csv, deletes from both articles.duckdb and filtered_articles.duckdb, and re-renders app/index.html. Editing the HTML or deleting DB rows is transient — main.py rebuilds filtered_articles.duckdb from scratch on every run, so only exclude_urls.csv makes an exclusion survive a rebuild. Verify after running main.py: no excluded URL should reappear in either store.

Making Changes

  • Edit main.py for code changes
  • Always call uv run to run the code
  • Lint and format with ruff check and ruff format
  • Always run pytest to test any edits
  • When editing markdown files, use markdownlint -c .markdownlint.jsonc <file> to lint the file
  • Edit app/index.html directly for content changes
  • Images should be placed in app/img/ directory

Local testing

Serve the site with:

python -m http.server 8000 --directory app

Use chrome-devtools-axi for all browser automation — do not use claude-in-chrome or playwright unless explicitly asked. It is installed as a project skill (npx -y chrome-devtools-axi <command>). Key patterns:

npx -y chrome-devtools-axi open http://localhost:8000   # navigate + snapshot
npx -y chrome-devtools-axi screenshot /tmp/out.png
npx -y chrome-devtools-axi console --type error         # JS errors only
npx -y chrome-devtools-axi stop

Inspect Alpine poster state from JS eval:

const data = Alpine.$data(document.querySelector('[x-data="poster"]'))
// data.idx       — current article index (0-based)
// data.articles  — full article array
// data.timer     — auto-advance interval id (clearInterval to stop)

Verify the rendered batch programmatically:

JSON.parse(document.getElementById('poster-articles').textContent).length  // article count

Deployment

Changes are automatically deployed via GitHub Pages when pushed to the main branch (.github/workflows/static.yml). The aggregation pipeline is rebuilt every 8 hours by .github/workflows/news-refresh.yml (cron 0 */8 * * *, plus workflow_dispatch for manual runs). See docs/ci.md for full workflow details.

Git Workflow

Use atomic commits with conventional commit format to ensure clean history and enable effective debugging with git bisect:

  • Make small, focused commits that do one thing well
  • Each commit should be self-contained and not break functionality
  • Use conventional commit format: <type>(<scope>): <description>
  • Common types: feat, fix, docs, style, refactor, test, chore
  • For conflicts or complex changes, break them into smaller atomic commits

Resources:

Context7

Always use Context7 MCP when I need library/API documentation, code generation, setup or configuration steps without me having to explicitly ask.

Libraries

  • alpinejs/alpine
  • astral-sh/ruff
  • astral-sh/uv
  • duckdb/duckdb-python
  • fastapi/fastapi
  • hunvreus/basecoat
  • jdx/mise
  • mrlesk/backlog.md
  • pbakaus/impeccable
  • tailwindlabs/tailwindcss
  • websites/duckdb_current
  • websites/taskfile_dev

<CRITICAL_INSTRUCTION>

BACKLOG WORKFLOW INSTRUCTIONS

This project uses Backlog.md MCP for all task and project management activities.

CRITICAL GUIDANCE

  • If your client supports MCP resources, read backlog://workflow/overview to understand when and how to use Backlog for this project.

  • If your client only supports tools or the above request fails, call backlog.get_workflow_overview() tool to load the tool-oriented overview (it lists the matching guide tools).

  • First time working here? Read the overview resource IMMEDIATELY to learn the workflow

  • Already familiar? You should have the overview cached ("## Backlog.md Overview (MCP)")

  • When to read it: BEFORE creating tasks, or when you're unsure whether to track work

These guides cover:

  • Decision framework for when to create tasks
  • Search-first workflow to avoid duplicates
  • Links to detailed guides for task creation, execution, and finalization
  • MCP tools reference

You MUST read the overview resource to understand the complete workflow. The information is NOT summarized here.

</CRITICAL_INSTRUCTION>