Update documentation and configuration for AWS credentials required for bioRxiv/medRxiv PDF downloads

Author: emptymalei

.env.example

@@ -1,5 +1,11 @@
 LLM_API_KEY=[YOUR_LLM_API_KEY]
 SEMANTIC_SCHOLAR_API_KEY=[YOUR_SEMANTIC_SCHOLAR_API_KEY]
 openalex_mailto=[YOUR_EMAIL; OPTIONAL_BUT RECOMMENDED]
+
+# AWS credentials — required for biorxiv / medrxiv full-text PDF download.
+# Since May 2025 biorxiv is protected by Cloudflare; paperscraper works around
+# this via the biorxiv TDM (Text & Data Mining) API backed by AWS S3.
+# Create a key in AWS IAM with the AmazonS3ReadOnlyAccess permission policy.
+# Without these keys the domain-expert full-text step will fail for biorxiv papers.
 AWS_ACCESS_KEY_ID=[YOUR_AWS_ACCESS_KEY_ID]
 AWS_SECRET_ACCESS_KEY=[YOUR_AWS_SECRET_ACCESS_KEY]

README.md

@@ -94,6 +94,33 @@ python -m swarm_notes.main
 
 Use the example in configs folder to create your own version.
 
+#### bioRxiv / medRxiv: AWS credentials for full-text PDF download
+
+Since May 2025 **biorxiv.org is protected by Cloudflare**, which blocks direct PDF
+downloads. When `paper_source` is `biorxiv` or `medrxiv` and `enable_domain_expert`
+is `true`, the pipeline uses the
+[paperscraper](https://github.com/jannisborn/paperscraper) library to fall back to the
+**biorxiv TDM (Text & Data Mining) API**, which serves PDFs from an AWS S3 bucket.
+
+This requires an AWS IAM key with **read-only S3 access**:
+
+1. Log in to the [AWS IAM console](https://console.aws.amazon.com/iam/).
+2. Create a new user (or access key for an existing user).
+3. Attach the `AmazonS3ReadOnlyAccess` managed policy.
+4. Generate an **Access Key ID** and **Secret Access Key**.
+5. Add them to your `.env` file (see `.env.example`):
+
+```bash
+AWS_ACCESS_KEY_ID=AKIA...
+AWS_SECRET_ACCESS_KEY=...
+```
+
+Without these credentials the domain-expert full-text step will be silently skipped
+for biorxiv/medrxiv papers.
+
+For CI/CD, add `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` as repository secrets
+(same place as `LLM_API_KEY`).
+
 ## CI/CD Setup
 
 ### Add the required secret

configs/config.example.yaml

@@ -21,4 +21,10 @@ site_description: "Automated research paper tracking and knowledge synthesis."
 # skills_dir: "skills"                       # Root directory for skills
 
 # Experimental domain expert agent (crawls HTML, consumes token quotas)
+# NOTE: when paper_source is "biorxiv" or "medrxiv", full-text PDF download
+# requires AWS credentials because biorxiv is Cloudflare-protected (since
+# May 2025). Set in your .env file:
+#   AWS_ACCESS_KEY_ID=...          # IAM key with AmazonS3ReadOnlyAccess
+#   AWS_SECRET_ACCESS_KEY=...
+# Without these the domain-expert step is skipped for biorxiv/medrxiv papers.
 enable_domain_expert: true

configs/config.neuro.yaml

@@ -23,4 +23,11 @@ active_skills:
 # skills_dir: "skills"                       # Root directory for skills
 
 # Experimental domain expert agent (crawls HTML, consumes token quotas)
+# NOTE: biorxiv PDFs are Cloudflare-protected (since May 2025). Full-text
+# analysis requires AWS credentials in your .env file so that paperscraper
+# can retrieve PDFs via the biorxiv TDM/S3 API:
+#   AWS_ACCESS_KEY_ID=...
+#   AWS_SECRET_ACCESS_KEY=...
+# Create an IAM key with AmazonS3ReadOnlyAccess. Without it the domain-expert
+# step will silently skip full-text analysis for biorxiv papers.
 enable_domain_expert: true