Add fastcrw-research skill for recall-first arXiv paper retrieval

[us]

What this adds

skills/fastcrw-research/SKILL.md, one file, no scripts and no new dependencies.

It documents four GET endpoints of the fastCRW Research API (paper search, single
paper, citation graph with references / citers / similar, and code search) and,
more importantly, the retrieval method that goes with them: exact-name query
decomposition as the base pass, plus routing a question to the citation graph when
the answer lives in a paper's bibliography rather than in a topical search
("what does X benchmark against", "who extends X", "which open model is best on Y").

Why this is not a second provider for a service an existing skill already reaches

AGENTS.md rules that out, so the skill is scoped and cross-referenced against what is
already here rather than overlapping it. Its own routing table sends the reader
elsewhere:

• PubMed, Europe PMC, bioRxiv, medRxiv, DOI/PMID resolution, open-access PDFs, full
  text: paper-lookup.
• A manuscript-ready evidence packet, evidence matrix, claim-to-source map:
  research-lookup.
• A PRISMA-style systematic review with screening and risk of bias:
  literature-review.

What is left is the part none of them target: recall-first enumeration over
arXiv-indexed work, and citation-graph traversal from a seed paper. The docs entry
says the same thing, so selection stays predictable.

Evidence

On ArXivQA (alphaXiv retriever-sandbox), 191 natural-language paper-retrieval
questions scored on recall, each provider queried on its own live deployed endpoint:

Provider	Recall
fastCRW Research API (this skill)	61.0%
Firecrawl Research Index	53.3%
Claude	45.4%
Parallel	44.3%
Exa	43.4%

Firecrawl's 53.3% is their own published figure from their Research Index launch
post; the rest is our run of the same 191 questions. Method and scope:
https://fastcrw.com/benchmarks/arxivqa-research-recall

The score is the agent plus this skill over live endpoints, with no pre-built paper
index and the ground truth hidden from the agent. Recall is not precision, and this
is a retrieval benchmark on arXiv-indexed questions, so it says nothing about
biomedical coverage, where paper-lookup is the right skill.

The comparison is offered as the reason this skill earns a slot, not as a claim
about exa-search or parallel-web as web toolkits: those cover extraction,
enrichment, and general web search that this skill does not attempt. The SKILL.md
itself does not repeat this table, it links the benchmark page.

Testing

• Every curl in the skill was run verbatim against the live API on 2026-08-29:
  all four returned HTTP 200 with results, including the intent requirement and the
  mode=references traversal.
• uv run skills-ref validate skills/fastcrw-research -> Valid skill.
• uv run --with pytest python -m pytest tests/_meta -q -> 10 passed, 1214 subtests.
• skill-scanner scan ./skills/fastcrw-research -> SAFE, 0 findings.
• The skill ships no scripts/, so no tests/fastcrw-research/ suite and no
  skill-requirements.toml entry are required.
• Repository lists updated: README skill count 163 to 164 (badge and three counts),
  one entry added to docs/skills.md under Research Methodology & Proposal Writing.

Notes

• Credential is FASTCRW_API_KEY, declared in compatibility and
  metadata.openclaw.envVars, with metadata.hermes.category: research. A new
  account includes 500 one-time credits and no card, so the skill is exercisable
  without a paid plan.
• The engine behind the API is open source under AGPL-3.0 at
  https://github.com/us/crw, so a lab that cannot send queries to a hosted service
  can run the same endpoints itself.
• The skill is honest about one live-graph limit: a seed paper with a very large
  citation count can return an empty citers list, and it tells the agent to fall
  back to the exact-name pass instead of reporting that nothing cites the paper.

Happy to adjust the naming, the scope boundaries, or the docs placement if you would
rather it sat somewhere else in the catalog.

[prashantpiyush1111]

The new skill is well-scoped and the documentation is clear. I did a careful pass over the new SKILL.md and the catalog updates; I don't see a blocking correctness issue in the current diff. One thing I would still verify before merge is the external benchmark/endpoint claims (61.0% ArXivQA recall, endpoint behavior, and the listed account/credit details), since the repository CI status exposed here is empty and those claims are not independently verifiable from the patch alone. If those checks were performed as described in the PR, the implementation looks reasonable to me.

[us]

Thanks, that is the right thing to push on, and checking it properly turned up a real defect in the skill rather than just confirming the number. Both are addressed.

The harness is now public. bench/arxivqa/ in the engine repo carries the scorer, the deterministic CLI the agent drives, the per-question prompt, and the stored 191-question run the published number is computed from:
https://github.com/us/crw/tree/main/bench/arxivqa

Two commands, no account needed for either:

python3 bench/arxivqa/score.py --selfcheck
# proves the scorer offline: id normalization, and an unanswered question scoring 0 rather than being skipped

python3 bench/arxivqa/score.py --results bench/arxivqa/runs/2026-06-20-hosted-191.jsonl
# mean recall over 191 questions: 0.6100  (61.0%)

The ground truth is downloaded from alphaXiv's retriever-sandbox on first use rather than vendored, so the score is checked against their data and not a copy of ours. Firecrawl's 53.3% is their own published figure, not our measurement of them.

The skill was incomplete, and I have pushed the fix to this PR. The base pass is two steps: the exact-name reframings, and then mode=references on the five ids the most reframings agreed on. Step A only described the first half, and the expansion was written up as a compare-against move. Measured on the first 12 benchmark questions with the query held fixed, so the expansion is the only variable:

	recall
search union alone	47.0%
with the top-5 reference expansion	66.5%

So an agent following the skill as it was written would not have reproduced the headline. That is exactly the class of problem your comment was pointing at, and it is worth more than the benchmark citation itself.

Endpoint behaviour is checkable in one call, and the four endpoints are in the open-core engine (crates/crw-server/src/routes/research.rs), so the whole benchmark also runs against a self-hosted crw serve with no key at all by setting FASTCRW_BASE_URL:

curl -s -H "Authorization: Bearer $FASTCRW_API_KEY" \
  "https://api.fastcrw.com/v1/search/research/papers?query=speculative%20decoding&k=5" | jq '.results[0].primaryId'
# "arxiv:2211.17192"

Account details: I removed the free-credit figure from the skill rather than restate it, because our own pricing page is currently inconsistent about it. It now links https://fastcrw.com/pricing so the number comes from the source instead of from me.

[prashantpiyush1111]

Thanks for addressing the verification points and for pushing the follow-up fix. The public benchmark harness and the reproducible scorer commands make the 61.0% figure much easier to verify, and I appreciate you explicitly identifying and fixing the incomplete retrieval flow rather than only defending the benchmark number. Removing the inconsistent free-credit figure is also the right call.

One remaining merge-gate item: GitHub currently reports that this branch has conflicts that must be resolved. Please rebase/merge the current base branch into the PR branch and resolve the conflicts, then push the updated branch. Once that is cleared, the updated skill and the validation claims can be re-checked against the resulting diff.

[us]

Rebased onto current main (cc37669) and pushed. The conflict was the README badge block: main moved the version badge to 2.65.0 while this branch moved the skills badge to 164. Resolution keeps both, 2.65.0 from main and 164 from here, and I did not touch pyproject.toml or plugin.json, so the collection version stays exactly as main set it.

Resolving it also surfaced a fifth skill count I had missed on the first pass, **163 Skills** - Extensive coverage in the What's Included list. All five now read 164, matching the 163 directories on main plus this one.

The diff against main is now just the three files:

README.md                        |  10 +--   (5 count bumps, nothing else)
docs/skills.md                   |   1 +     (one catalog entry)
skills/fastcrw-research/SKILL.md | 150 +++   (new)

Re-checked after the rebase, on the rebased tree:

uv run skills-ref validate skills/fastcrw-research   -> Valid skill
uv run --with pytest python -m pytest tests/_meta -q -> 10 passed, 1214 subtests passed
skill-scanner scan ./skills/fastcrw-research         -> SAFE, 0 findings

The skill still ships no scripts/, so no tests/fastcrw-research/ suite and no skill-requirements.toml entry are needed. The benchmark claims are unchanged and still checkable with the two scorer commands in the previous comment.