basted-pocket

Design

Okay, this is a fantastic direction! A view-only static site generated by GitHub Actions, with the potential for "editing" by directly modifying the source Markdown and an optional PAT-enabled edit mode in the app later.

Here's a detailed design specification:

Project Title: "LinkHarbor" - A GitOps-Powered Read-It-Later Service

1. Core Objective: To create a system where users can submit URLs via a central Markdown file in a GitHub repository. A GitHub Action will process these URLs, enrich them with scraped content and LLM-generated metadata (summaries, auto-tags), and publish this enriched data as a static website (via GitHub Pages) for easy, view-only consumption.

2. System Components:

• A. GitHub Repository (Public):

  • links.md: The primary input file where users add URLs and optional initial notes/tags.
  • config/tag_definitions.json: Definitions and descriptions for tags to guide LLM auto-tagging.
  • config/prompts.json: (Optional) Centralized store for LLM prompt templates.
  • .github/workflows/process_links.yml: The GitHub Actions workflow definition.
  • scripts/: Directory for helper scripts used by the GitHub Action (e.g., Python/Node.js for parsing, scraping, LLM calls).
  • site_template/: Directory containing the HTML, CSS, and JavaScript templates for the static GitHub Pages site.
  • docs/: (Standard directory for GitHub Pages) This is where the generated static site files will be placed by the Action.

• B. GitHub Actions Workflow (process_links.yml):

  • The automation engine.

• C. Static GitHub Pages Site:

  • The view-only frontend, displaying the enriched articles.

3. Data Flow & Processing:

Input Stage (Manual):

1. Users (you and your wife) edit links.md directly on GitHub or by cloning, editing, and pushing.
   • Format in links.md:

     # LinkHarbor Entries
     
     - https://example.com/article1 #initial_tag @note:Read for project X
     - [Optional Title for URL](https://example.com/article2)
     - https://blog.example.com/long-post #to_discuss

   • A unique identifier for each link will be its canonicalized URL.

Processing Stage (GitHub Actions Workflow - process_links.yml):

• Trigger: On push to main branch affecting links.md (or config/ files).
• Job Steps:
  1. Checkout Code: actions/checkout@v3
  2. Setup Environment: Setup Python/Node.js, install dependencies from requirements.txt or package.json.
  3. Load Configuration:
     • Read config/tag_definitions.json.
     • Read config/prompts.json (if used).
  4. Parse links.md:
     • A script (scripts/parse_markdown.py or .js) will:
       • Read links.md.
       • Extract all URLs, any user-provided inline tags (e.g., #tag), and notes (e.g., @note: ...).
       • Canonicalize URLs (resolve redirects, normalize).
       • Generate a unique ID for each article (e.g., SHA256 hash of the canonical URL).
       • Output: A list of raw article objects: [{id: "...", original_url: "...", canonical_url: "...", user_tags: ["..."], user_notes: "..."}].
  5. Load Existing Processed Data (for diffing/caching - optional optimization):
     • Attempt to read a processed_data_cache.json from a previous run (could be an artifact downloaded, or a file in a temporary branch). This helps avoid re-processing unchanged URLs. For simplicity in v1, we can re-process all non-archived links.
  6. Process Each Article (Iterate through parsed links):
     • For each unique canonical_url:
       • a. Skip if Already Processed & Unchanged (Optimization): If links.md entry for this URL hasn't changed and content fetch isn't forced, potentially skip.
       • b. Scrape Content (scripts/scraper.py or .js):
         • Input: canonical_url.
         • Libraries: requests, BeautifulSoup4, trafilatura (Python) or node-fetch, jsdom, @mozilla/readability (Node.js).
         • Output: fetched_title, main_text_content, main_html_content (simplified), publication_date, author, key_image_url, json_ld_data (especially for recipes).
         • Handle errors gracefully.
       • c. LLM Enrichment (scripts/llm_enricher.py or .js):
         • Input: main_text_content, tag_definitions.
         • Requires OPENAI_API_KEY (or other LLM provider key) from GitHub Secrets.
         • i. Auto-Tagging:
           • Prompt: Use tag_definitions to guide LLM.
           • Output: auto_tags: ["tag1", "tag2"].
         • ii. Summarization:
           • Prompt: Generate a concise summary.
           • Output: summary: "...".
         • iii. Keyword Extraction:
           • Prompt: Extract key topics/keywords.
           • Output: keywords: ["kw1", "kw2"].
         • iv. Read Time Estimation:
           • Calculate based on word count of main_text_content.
           • Output: read_time_minutes: N.
         • Handle LLM API errors.
       • d. Aggregate Data: Combine original data, scraped data, and LLM outputs into a single JSON object for this article.

         // Example article_data.json structure
         {
           "id": "sha256_of_canonical_url",
           "original_url": "...",
           "canonical_url": "...",
           "user_tags": [],
           "user_notes": "...",
           "time_added_to_links_md": "timestamp_from_git_commit", // Approximate
           "fetched_title": "...",
           "main_text_content": "...", // Could be extensive
           "main_html_content": "...", // For reader view rendering
           "publication_date": "...",
           "author": "...",
           "key_image_url": "...",
           "json_ld_recipes": [{}], // Array of Recipe JSON-LD objects if found
           "auto_tags": ["...", "..."],
           "summary": "...",
           "keywords": ["...", "..."],
           "read_time_minutes": 0,
           "processing_timestamp": "iso_timestamp_of_this_run",
           "status": "processed" // or "error_scraping", "error_llm"
         }

  7. Consolidate Processed Data:
     • Collect all processed article JSON objects into a single list.
     • Save this list as build_output/all_articles_data.json.
     • (Optional) Save individual article JSONs: build_output/articles/{id}.json.
  8. Generate Static Site (scripts/generate_site.py or .js):
     • Input: build_output/all_articles_data.json, site_template/.
     • Templating Engine: Jinja2 (Python), Nunjucks/Handlebars (Node.js).
     • Output Files (to be placed in docs/ directory):
       • docs/index.html: Main listing page with search, filters (by tag).
       • docs/articles/{id}.html: Individual article pages (reader view).
       • docs/data/all_articles_data.json: A copy of the consolidated data for client-side JavaScript use on the site.
       • docs/assets/: CSS, JavaScript for the site.
       • docs/images/: (Optional) If key images are downloaded and self-hosted.
  9. Deploy to GitHub Pages:
     • Use an action like actions/deploy-pages@v1 or peaceiris/actions-gh-pages@v3.
     • Configure it to deploy the contents of the docs/ directory.

Output Stage (GitHub Pages Site):

• URL: https://<username>.github.io/<repository-name>/
• Features:
  • Homepage (index.html):
    • List of all processed articles (title, summary, tags, key image).
    • Client-side search bar (searches titles, summaries, keywords from the fetched docs/data/all_articles_data.json).
    • Client-side filtering by tags.
    • Sort options (date added, publication date).
    • Links to individual article pages.
  • Article Page (articles/{id}.html):
    • Clean reader view (using main_html_content or rendering main_text_content nicely).
    • Displays title, author, publication date, full summary, tags, keywords.
    • Link to original URL.
    • If recipe data exists, display it formatted.

4. Key Files & Structures:

• links.md: As described. Simple list of URLs with optional inline tags/notes.
• config/tag_definitions.json:

  {
    "tags": [
      {"name": "tech", "description": "Articles about software, hardware, internet.", "keywords_hint": ["code", "server"]},
      {"name": "ai", "description": "Artificial intelligence, machine learning.", "keywords_hint": ["neural network", "LLM"]},
      {"name": "project_idea", "description": "Potential ideas for new projects."},
      {"name": "long_read", "description": "Articles that require significant time to read."}
    ]
  }

• site_template/:
  • base.html: Base HTML template with common head, nav, footer.
  • index_template.html: Template for the main listing page.
  • article_template.html: Template for the individual article reader view.
  • assets/style.css: Site styling.
  • assets/script.js: Client-side JavaScript for search, filtering on index.html.
• Generated docs/data/all_articles_data.json: The primary data source for the live static site's JavaScript.

5. LLM Integration:

• Use OpenAI API (or other provider like Anthropic).
• API Key stored as a GitHub Secret (OPENAI_API_KEY).
• Prompts designed for:
  • Summarization (e.g., "Summarize this article in 3 bullet points for a busy professional.")
  • Tagging (e.g., "Based on the following article text and tag definitions, assign up to 3 relevant tags: ... Definitions: ... Text: ...")
  • Keyword extraction.
• Error handling and retries for API calls.

6. Future Considerations (Post-MVP):

• Edit Mode in Static App:
  • If a PAT is provided by the user (and securely handled by the app):
    • App could use GitHub API to update links.md (e.g., adding a new URL, or adding/removing a user tag directly in the Markdown via API).
    • This would trigger the Action to re-process.
• Archiving/Deleting: How to mark links in links.md as "archived" or "to be deleted" so the Action can filter them out or remove their generated pages. (e.g., a special tag #archive_me or moving to a ## Archived Links section in links.md).
• Local Image Caching: Action could download key images and serve them from GitHub Pages to avoid hotlinking and improve privacy/reliability.
• Incremental Processing: More robust logic to only re-process genuinely new or changed links, perhaps by hashing the relevant line in links.md.
• Advanced Search: Using a client-side search library like Lunr.js or Fuse.js for better search on the static site.
• Error Reporting: If an Action run fails, how are users notified? (GitHub's built-in notifications for workflow failures).

7. Security & Privacy:

• GitHub repository is public; links.md and generated site are public.
• LLM API keys are stored as GitHub Secrets.
• No sensitive user data stored beyond the URLs they choose to save.
• If images are proxied/cached by the Action, it adds a layer of privacy from the original site trackers.

This design provides a robust, maintainable, and highly automated system leveraging the power of GitOps and GitHub's ecosystem. The view-only static site is a great starting point for consumption.

---

Dev Plan

Okay, this is a great evolution of the idea! Using Bun.ts for scripting and the Gemini API for LLM tasks is very modern and efficient. The static site as a view-only output artifact is a clean and scalable approach.

Here's a step-by-step development plan for "LinkHarbor" using this stack:

Project Title: LinkHarbor (Bun.ts + Gemini Edition)

Core Technologies:

• Runtime & Scripting: Bun with TypeScript (.ts files)
• LLM: Google Gemini API (via @google/genai)
• Input: links.md (Markdown file in GitHub repo)
• Automation: GitHub Actions
• Output: Static HTML/CSS/JS site deployed to GitHub Pages

---

Development Plan:

Phase 1: Foundation & Local Processing

Step 1: Reverse Engineer Pocket Export & Bootstrap links.md * Goal: Create an initial links.md from your existing Pocket data. * Tasks: 1. Obtain your Pocket export file (likely ril_export.html or a JSON if available). 2. Analyze its structure. If HTML, it's a list of <a> tags. Tags might be in data- attributes or nearby. 3. Write a one-off bun.ts script (scripts/importPocket.ts): * Takes the Pocket export file as input. * Parses it (using a DOM parser like node-html-parser if HTML, or JSON.parse if JSON). * Extracts URLs, titles (if available), and existing tags. * Formats each entry for links.md: markdown - [Optional Pocket Title](https://example.com/url1) #pocket_tag1 #pocket_tag2 @note:Added_to_Pocket_on_YYYY-MM-DD * Outputs the content for links.md. 4. Manually create the initial links.md in your repository with this output. * Deliverable: An initial links.md file in the repository.

Step 2: Project Setup & Basic Repository Structure * Goal: Establish the project's file and directory structure. * Tasks: 1. Create a new public GitHub repository. 2. Initialize Bun: bun init -y (creates package.json, tsconfig.json, etc.). 3. Install initial dependencies: bun add @google/genai @types/bun marked node-html-parser (or similar for DOM parsing if needed). 4. Create directory structure: * .github/workflows/ * config/ * scripts/ * site_template/ * site_template/assets/ (for css, js) * docs/ (GitHib Pages output, add .gitignore for its contents initially) 5. Add your bootstrapped links.md to the root. 6. Create config/tag_definitions.json (start with a few example tags). json // config/tag_definitions.json { "tags": [ {"name": "tech", "description": "Articles about software, hardware, internet technology."}, {"name": "ai", "description": "Focus on Artificial Intelligence, machine learning, and related topics."} ] } 7. Create a basic .gitignore (e.g., for node_modules, .env, build_output/, docs/ initially). * Deliverable: A GitHub repository with the basic structure and initial configuration files.

Step 3: Markdown Parsing Script (scripts/parseMarkdown.ts) * Goal: Programmatically read links.md and extract link information. * Tasks: 1. Create scripts/parseMarkdown.ts. 2. Use a Markdown parser (e.g., marked library, or simpler regex if the format is very strict). 3. Function to parse links.md: * Identifies lines starting with - (list items). * Extracts URL, optional title from [title](url) format. * Extracts inline tags (#tag). * Extracts inline notes (@note:text). * Canonicalizes URLs (basic normalization, actual redirect resolution comes later). * Generates a unique ID (e.g., SHA256 of the canonical URL). 4. Output an array of objects, e.g.: typescript interface ParsedLink { id: string; original_url: string; canonical_url: string; // Initially same as original_url user_title?: string; user_tags: string[]; user_notes?: string; // Placeholder for commit timestamp later time_added_to_links_md?: string; } * Deliverable: A Bun script that can read links.md and output structured data.

Step 4: Web Scraping Script (scripts/scrapeUrl.ts) * Goal: Fetch and extract content from a single URL. * Tasks: 1. Create scripts/scrapeUrl.ts. 2. Use Bun's built-in fetch or node-fetch. 3. DOM parsing: node-html-parser or jsdom (jsdom is heavier but handles more complex JS-driven sites if needed, though for a GitHub Action, simpler is better). 4. Content extraction: Use @mozilla/readability (ported or available for Node/Bun) or trafilatura (if a Bun-compatible version/wrapper exists, or call Python script as a fallback). 5. Function scrapeUrl(url: string): Promise<ScrapedData>: * Fetches the URL, handles redirects (update canonical_url). * Extracts: fetched_title, main_text_content, main_html_content (simplified for reader view), publication_date (heuristics), author (heuristics), key_image_url (og:image, largest relevant image), json_ld_data (find <script type="application/ld+json">). * Implement robust error handling. typescript interface ScrapedData { canonical_url: string; fetched_title?: string; main_text_content?: string; main_html_content?: string; publication_date?: string; author?: string; key_image_url?: string; json_ld_objects?: any[]; // Parsed JSON-LD error?: string; } * Deliverable: A Bun script that can scrape a URL and return structured content.

Step 5: LLM Enrichment Script with Gemini (scripts/enrichWithGemini.ts) * Goal: Use Gemini API to generate summaries, tags, and keywords. * Tasks: 1. Create scripts/enrichWithGemini.ts. 2. Import @google/genai. 3. Function enrichData(text_content: string, tag_definitions: any): Promise<EnrichedData>: * Initialize GoogleGenAI with process.env.GEMINI_API_KEY. * Auto-Tagging: * Construct prompt using text_content and tag_definitions.json. * Prompt Gemini to return a list of tags (e.g., "Return as a comma-separated list: TAG1,TAG2"). Parse the response. * Summarization: * Prompt for a concise summary. * Keyword Extraction: * Prompt for main keywords. * Calculate read_time_minutes (simple word count / average reading speed). * Handle API errors, rate limiting (basic retry). typescript interface EnrichedData { auto_tags?: string[]; summary?: string; keywords?: string[]; read_time_minutes?: number; llm_error?: string; } * Deliverable: A Bun script that uses Gemini to enrich text content.

Phase 2: Automation & Site Generation

Step 6: Orchestration Script (scripts/mainProcessor.ts) * Goal: Tie together parsing, scraping, and enrichment for all links. * Tasks: 1. Create scripts/mainProcessor.ts. 2. Import functions from parseMarkdown.ts, scrapeUrl.ts, enrichWithGemini.ts. 3. Load config/tag_definitions.json. 4. Main function: * Calls parseMarkdown() to get links. * Iterates through parsed links: * Calls scrapeUrl() for each. * If scrape successful, calls enrichWithGemini() with main_text_content. * Merges all data (parsed, scraped, enriched) into a final article object (structure defined in previous design spec). * (Future: Add logic to get time_added_to_links_md from Git commit history for that line). * Collects all final article objects into a list. * Saves the consolidated list to build_output/all_articles_data.json. * (Optional) Saves individual article JSONs to build_output/articles/{id}.json. * Deliverable: A master script that processes links.md and produces consolidated JSON data.

Step 7: Static Site Generator Script (scripts/generateSite.ts) * Goal: Create HTML files for GitHub Pages from the processed JSON data. * Tasks: 1. Create scripts/generateSite.ts. 2. Choose a templating engine compatible with Bun/Node.js (e.g., nunjucks, handlebars, or simple string replacement). Install if needed: bun add nunjucks @types/nunjucks. 3. Create basic HTML templates in site_template/: * base.html (common layout) * index_template.html (for article listing) * article_template.html (for single article view) * Basic site_template/assets/style.css and site_template/assets/script.js (for client-side search/filter later). 4. Main function: * Reads build_output/all_articles_data.json. * Renders docs/index.html using index_template.html and the list of articles. * For each article, renders docs/articles/{id}.html using article_template.html. * Copies build_output/all_articles_data.json to docs/data/all_articles_data.json (for client-side JS). * Copies site_template/assets/ to docs/assets/. * Deliverable: A script that generates the docs/ folder for GitHub Pages.

Step 8: GitHub Actions Workflow (.github/workflows/process_links.yml) - Initial * Goal: Automate the entire processing pipeline on code push. * Tasks: 1. Create .github/workflows/process_links.yml. 2. Define trigger: on: push: paths: ['links.md', 'config/**'] branches: [main] 3. Job steps: * actions/checkout@v4 * Setup Bun: uses: oven-sh/setup-bun@v1 * Install dependencies: run: bun install * Run main processor: run: bun run scripts/mainProcessor.ts * Pass GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} as an environment variable. * Run site generator: run: bun run scripts/generateSite.ts * Temporary: Upload docs/ and build_output/ as artifacts for inspection: actions/upload-artifact@v3. 4. Add GEMINI_API_KEY as a repository secret in GitHub settings. * Deliverable: A GitHub Action that runs the processing pipeline and produces artifacts. Test by pushing changes to links.md.

Phase 3: Deployment & Refinement

Step 9: GitHub Pages Deployment in Action * Goal: Automatically deploy the generated docs/ folder to GitHub Pages. * Tasks: 1. Modify process_links.yml: * Add permissions for pages: write and id-token: write. * Add actions/configure-pages@v4. * Replace artifact upload with actions/upload-pages-artifact@v3 (uploads docs/). * Add actions/deploy-pages@v4 to deploy the artifact. 2. Configure GitHub Pages settings in the repository to deploy from the gh-pages branch (or from Actions artifacts). * Deliverable: The static site is automatically deployed and accessible online.

Step 10: Enhance Static Site (Client-Side Features) * Goal: Make the static site more usable. * Tasks: 1. In docs/assets/script.js (or site_template/assets/script.js): * Fetch docs/data/all_articles_data.json. * Implement client-side search (filter articles based on input). * Implement client-side filtering by tags. * Implement sorting options. 2. Improve CSS in docs/assets/style.css for better presentation. * Deliverable: A more interactive and polished view-only static site.

Step 11: Refinements, Error Handling, and Optimizations * Goal: Make the system more robust and efficient. * Tasks: * Error Handling: Improve error handling in all scripts (scraper, LLM calls, file operations). Log errors clearly in Action output. * Logging: Add more detailed logging throughout the Action scripts. * Idempotency/Caching (Optimization): * Store a hash or last modified timestamp for each URL processed. * In mainProcessor.ts, only scrape/enrich URLs that are new or whose links.md entry has changed (more complex, could use Git history to check line changes). * This reduces redundant processing and LLM calls. * Archiving/Deletion Logic: * Define a convention in links.md (e.g., ## Archived section, or #archive tag). * Update parseMarkdown.ts to identify these. * Update generateSite.ts to exclude archived items or move them to a separate page. * Prompt Engineering: Refine Gemini prompts for better summaries, tags, and keyword quality. * Rate Limiting (LLM): Implement basic delays between LLM calls if processing many articles in one run. * Deliverable: A more stable, efficient, and feature-complete system.

Step 12: Documentation * Goal: Document the project for yourself and potential collaborators. * Tasks: 1. Create/update README.md: * Project purpose. * How to add links to links.md. * Explanation of the GitHub Actions workflow. * How to set up GEMINI_API_KEY secret. * Structure of config/tag_definitions.json. * Deliverable: Comprehensive project documentation.

---

This plan breaks the project into manageable phases and steps, focusing on using Bun.ts and Gemini as requested. Each step builds upon the previous one, leading to the desired automated, GitOps-driven read-it-later service. Good luck!