Contributions are always welcome! Use the Scraping Configuration help section to get started and stop by the Discord #scrapers channel with any questions.
If possible, XPath or JSON based yml scrapers are always preferred over python scrapers unless explicitly required
The scrapers in this repository can be validated against a schema and checked for common errors.
Deno is used as a drop-in, sandboxed NodeJS alternative
# check all scrapers
deno run -R=scrapers -R="validator\scraper.schema.json" validate.js
# check specific scraper
deno run -R=scrapers -R="validator\scraper.schema.json" validate.js scrapers/foo.yml scrapers/bar.ymlDeno asks for env and sys permissions from chalk
Instead of Deno, Docker can be used to run the validator
docker run --rm -v .:/app denoland/deno:distroless run -R=/app/ -E /app/validate.js --ci