Skip to content

SKILL.md

Latest commit

 

History

History
233 lines (184 loc) · 12.3 KB

SKILL.md

File metadata and controls

233 lines (184 loc) · 12.3 KB
Raw
name Web2Skill
description Convert one public website URL or an explicit batch of public URLs into a reusable skill zip backed by rendered HTML snapshots and a bounded JSONL retrieval index. Use to discover a documentation directory from one URL, crawl a supplied URL set sequentially, generate a source profile, or package indexed web content as a specialized skill.

Web2Skill

Use three deterministic stages. Keep every agent-created intermediate file and the final zip under the writable task directory provided by the system prompt. Run all scripts with Python 3. Do not install or upgrade packages or browser binaries at runtime.

Let <skill-dir> be the skill base directory supplied by the skill loader immediately above these instructions. Use that directory exactly as supplied. Execute bundled scripts in place with python3 <skill-dir>/scripts/...; never copy them into the task directory or a temporary directory.

Let <task-dir> be the writable task directory supplied by the system prompt. Use it exactly as supplied for every intermediate artifact, model-authored JSON file, and final zip.

Choose create or update mode

Use create mode when no existing skill zip is supplied.

Use update mode only when the request explicitly supplies an existing generated skill zip path and the required unchanged skill name. Do not extract or inspect the zip with ad hoc commands. The update-mode Stage 2 command validates the archive and stages its old index and HTML under <task-dir>/existing/.

Update mode is a current-scope rebuild, not an append-only merge. Stage 1 must rebuild the URL list using the original input mode: rediscover the current directory from one entry URL, or validate, normalize, and deduplicate only the current explicit batch when two or more URLs are supplied. For each current URL, Stage 2 may reuse a valid matching page from the staged cache. Pages found only in the old zip are not copied into the new crawl, so pages outside the rebuilt scope disappear from the updated skill. Cached Yuque error pages and advertisement-only pages are rejected and fetched again.

Stage 1: Prepare the URL list

Always run <skill-dir>/scripts/prepare_urls.py first.

  • When the user supplies exactly one URL, the script opens that page with Playwright, detects its directory or navigation tree, and writes the start page plus all discovered in-scope page URLs.
  • When the user supplies two or more URLs, the script skips all network discovery. It only validates, normalizes, deduplicates, and writes the supplied URLs.
  • The output is always a UTF-8 text file with one URL per line.
python3 <skill-dir>/scripts/prepare_urls.py \
  --out <task-dir>/crawl/url-list.txt \
  "https://example.com/docs"
python3 <skill-dir>/scripts/prepare_urls.py \
  --out <task-dir>/crawl/url-list.txt \
  "https://example.com/page-a" \
  "https://example.com/page-b"

Do not manually create the URL-list file and do not run directory discovery for a batch supplied by the user.

Directory navigation and each directory-collection pass use fixed 10-minute limits. A failed navigation is retried once. Directory scroll passes, convergence, and any truncation reason are appended to <task-dir>/crawl/crawl.log; they are not added to the URL-list file or the retrieval index.

For ChatWiki Docs (help.chatwiki.com), use the Docusaurus sitemap rather than the currently visible sidebar. Rebuild sitemap paths on the supplied origin because the sitemap publishes a placeholder host, and preserve the language selected by the supplied start URL.

For KanCloud books, read the complete summary tree from application/payload+json before using rendered catalog links. Treat a result containing only the supplied page as a preparation failure; do not continue to build a one-page skill.

Stage 2: Crawl the prepared URLs

Run <skill-dir>/scripts/crawl_urls.py with the URL-list file from Stage 1.

python3 <skill-dir>/scripts/crawl_urls.py \
  --url-list <task-dir>/crawl/url-list.txt \
  --out-dir <task-dir>/crawl

In update mode, supply the existing skill zip and required unchanged name:

python3 <skill-dir>/scripts/crawl_urls.py \
  --url-list <task-dir>/crawl/url-list.txt \
  --out-dir <task-dir>/crawl \
  --existing-skill <existing-skill.zip> \
  --expected-name <existing-skill-name>

The crawler has intentionally fixed behavior:

  • Crawl URLs sequentially with no concurrency.
  • Use Playwright and a fixed 60-second page timeout.
  • Retry one time after a timeout, browser network error, HTTP 429 or 5xx response, empty rendered body, Yuque error page, or advertisement-only page.
  • Stop after four consecutive final timeouts and skip the remaining URLs. A success or a non-timeout failure resets the timeout streak.
  • Apply built-in body selectors for ChatWiki Docs, Yuque, Feishu, OpenClaw Docs, Alibaba Cloud Help, KanCloud, and WeChat Official Account articles; use the rendered page body as the fallback.
  • For Feishu, retain the longest stable body snapshot when the final body is empty or shorter.
  • Save cleaned rendered HTML under <task-dir>/crawl/html/, removing recognized advertisement nodes.
  • In update mode, check each current normalized URL against the staged existing index. Reuse only a complete readable record whose HTML is neither a Yuque error page nor an advertisement-only page. Remove recognized advertisement nodes from an otherwise reusable cached snapshot. Write reused records and HTML into the new crawl output one page at a time; never prefill the new crawl with the complete old index or HTML directory.
  • If every current URL is reused, skip launching Chromium and still write a complete crawl log and fresh index.
  • For www.yuque.com, retry a rendered error page once, then record it as yuque_error_page and omit it from the HTML snapshots and retrieval index.
  • Retry a page whose #main is empty and whose remaining content contains only recognized advertisements. If it remains advertisement-only, record it as a crawl failure and omit it from the HTML snapshots and retrieval index.
  • When different prepared URLs redirect to the same final URL, capture and index that page once. Record the other prepared URLs as redirect duplicates in crawl coverage; they are not crawl failures.
  • Extract keywords with jieba from the title, description, and selected body, then merge them with the page's original metadata keywords.
  • When the final index contains at least four pages, including reused pages, remove cross-page high-frequency noise terms unless they occur in the page title or description, then keep at most 12 keywords per page.
  • Append one successful page object per line to <task-dir>/crawl/index.jsonl.
  • Append immediate structured progress, retry, timeout-stop, and error events to the same <task-dir>/crawl/crawl.log created during Stage 1.

Use debug mode only when validating the workflow. It processes at most the first five URLs:

python3 <skill-dir>/scripts/crawl_urls.py \
  --url-list <task-dir>/crawl/url-list.txt \
  --out-dir <task-dir>/crawl-debug \
  --debug

Do not pass browser, concurrency, wait, retry, depth, link-scope, or timeout options. Those controls are not part of this workflow.

Never edit or delete the generated URL list, JSONL index, crawl log, or rendered HTML snapshots by hand. After crawl validation succeeds, never rerun URL preparation or crawling because a downstream stage fails. If crawl validation fails, rerun the crawler at most once without deleting artifacts; if the same error repeats, return that error instead of restarting the workflow.

Validate the completed crawl with the bundled helper:

python3 <skill-dir>/scripts/validate_crawl.py \
  --index <task-dir>/crawl/index.jsonl

The helper resolves every crawl-index html_path relative to the directory containing index.jsonl, never relative to the process working directory. It returns a bounded completion result containing the latest crawl_urls run.done counts, a bounded failure summary, a bounded redirect-duplicate summary, and a bounded Yuque error-page-skip summary. Do not open, list, print, cat, or otherwise load crawl.log through model file tools or ad hoc commands. Use only the helper output when evaluating crawl completion, failures, redirect duplicates, or skipped Yuque error pages.

Stage 3: Build the specialized skill

Run the deterministic metadata-outline helper. Do not open, list, or read index.jsonl through model file tools.

python3 <skill-dir>/scripts/metadata_outline.py \
  --index <task-dir>/crawl/index.jsonl

The helper returns at most 60 compact page records. It allocates slots in proportion to each source site's successful page count, preserves source-site coverage when the limit permits, and samples evenly within each site instead of taking only its first pages. Use only the returned outline and exact URLs to create one UTF-8 JSON metadata file at <task-dir>/skill-metadata.json.

Read references/metadata.md for the schema and limits. The model must provide the skill identity and source profile; the build script does not infer them. If outline_truncated is true, describe represented themes without claiming exhaustive index coverage. Do not infer negative coverage from topics missing in the outline.

In update mode, set the top-level name in <task-dir>/skill-metadata.json to the exact required unchanged skill name supplied by the system prompt. Copy that value verbatim; do not infer, normalize, translate, or regenerate it from the current outline. Regenerate all other metadata fields from the current final index using the normal schema.

If build validation rejects the metadata file, correct only <task-dir>/skill-metadata.json and rerun the build stage. For an update-mode metadata.name mismatch, replace only that field with the exact required unchanged name before rerunning the build. Do not repeat URL preparation or crawling when their validated outputs are already present.

The build stage also reads the last crawl_urls run.done event from <task-dir>/crawl/crawl.log, validates its counts against index.jsonl, and writes a deterministic reused, succeeded, failed, redirect-duplicate, Yuque-error-page-skip, and timeout-skipped coverage note into the generated Skill. Model-authored coverage_notes supplement this deterministic crawl boundary and cannot replace or hide it.

Use the metadata name value as <skill-name> and build the zip:

python3 <skill-dir>/scripts/build_skill.py \
  --index <task-dir>/crawl/index.jsonl \
  --metadata <task-dir>/skill-metadata.json \
  --zip-out <task-dir>/generate_skill/<skill-name>.zip

In update mode, pass the required unchanged name:

python3 <skill-dir>/scripts/build_skill.py \
  --index <task-dir>/crawl/index.jsonl \
  --metadata <task-dir>/skill-metadata.json \
  --expected-name <existing-skill-name> \
  --zip-out <task-dir>/generate_skill/<existing-skill-name>.zip

The generated skill contains:

<skill-name>/
|-- SKILL.md
|-- agents/
|   `-- openai.yaml
|-- references/
|   |-- web-index.jsonl
|   `-- html/
|       `-- *.html
`-- scripts/
    |-- search_index.py
    `-- fetch_rendered_html.py

search_index.py is the bounded local retrieval helper. fetch_rendered_html.py fetches one current rendered page only when the saved snapshots are insufficient or the user explicitly requests current content.

Validation and completion

Before returning success:

  1. Confirm the crawl-validation helper reports status: complete with nonzero url_count and index_rows. Require run_done.requested == url_count, run_done.reused + run_done.succeeded == index_rows, html_paths_checked == index_rows, failure_summary.count == run_done.failed, duplicate_summary.count == run_done.duplicate_final_urls, and error_page_skip_summary.count == run_done.skipped_error_pages.
  2. Confirm the metadata-outline helper reports status: complete, at least one outline item, and no more than 60.
  3. Confirm the build command succeeds. Its deterministic input validation and packaging create the required Skill contents at <task-dir>/generate_skill/<skill-name>.zip.
  4. Immediately return only the final zip path required by the system prompt. Do not call ls or reopen the zip, index, HTML snapshots, or crawl.log after a successful build. Inspect them only to diagnose a reported script error.