docs: add CRW as Firecrawl-compatible alternative for web search#152
docs: add CRW as Firecrawl-compatible alternative for web search#152us wants to merge 2 commits intou14app:mainfrom
Conversation
Add documentation for CRW (https://github.com/us/crw), an open-source Firecrawl-compatible web scraper that can be used as a drop-in replacement. Includes setup instructions for both self-hosted and cloud (fastcrw.com) options.
There was a problem hiding this comment.
Pull request overview
Adds documentation for using CRW as a Firecrawl-compatible alternative web search provider, including example configuration for self-hosted and hosted deployments.
Changes:
- Document CRW as a Firecrawl-compatible web scraper and add setup steps (self-hosted + cloud).
- Add CRW to the README’s supported web search providers list.
- Add
env.tplcomments to advertise CRW usage viaFIRECRAWL_API_BASE_URL.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
env.tpl |
Adds inline comments showing CRW endpoints as alternatives for FIRECRAWL_API_BASE_URL. |
README.md |
Adds CRW to the supported providers list and introduces a new “Using CRW as a Firecrawl Alternative” section with setup examples. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
README.md
Outdated
| [CRW](https://github.com/us/crw) is an open-source, Firecrawl-compatible web scraper that you can self-host or use via the cloud ([fastcrw.com](https://fastcrw.com)). Since CRW is a drop-in replacement for Firecrawl, you can use it as your search provider without any code changes — just set the `FIRECRAWL_API_BASE_URL` environment variable. | ||
|
|
There was a problem hiding this comment.
The wording here implies that setting FIRECRAWL_API_BASE_URL alone is sufficient in all modes. In this codebase, FIRECRAWL_API_BASE_URL only affects the server-side/proxy path (/api/search/firecrawl); in local mode the base URL is taken from the Firecrawl settings (firecrawlApiProxy). Please clarify which mode(s) this applies to and where users should set the URL in local mode.
| ```bash | ||
| FIRECRAWL_API_BASE_URL=http://localhost:3002 | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Using http://localhost:3002 will not work when Deep Research is running in a Docker/container environment ("localhost" will resolve inside the app container, not to the host/another container). Consider adding a short note with an example of a reachable URL for common setups (e.g., host.docker.internal, docker-compose service name, or shared network).
| > **Note (Docker/container setups):** If Deep Research is running in a container, `localhost` will resolve inside that container. Use a URL that is reachable from the Deep Research container, for example: | |
| > | |
| > - If CRW is running on the host with `-p 3002:3002`: `FIRECRAWL_API_BASE_URL=http://host.docker.internal:3002` | |
| > - If CRW is another container on the same Docker network or in `docker-compose`: `FIRECRAWL_API_BASE_URL=http://crw:3002` (or your CRW service name) |
| # Self-hosted: FIRECRAWL_API_BASE_URL=http://localhost:3002 | ||
| # Cloud: FIRECRAWL_API_BASE_URL=https://fastcrw.com |
There was a problem hiding this comment.
These examples use localhost, which can be misleading for Docker deployments (the app container won't be able to reach a CRW instance running on the host or in a different container via localhost). Consider adjusting the comments to mention using a host/container-reachable address (e.g., host.docker.internal or a compose service name) depending on the deployment topology.
| # Self-hosted: FIRECRAWL_API_BASE_URL=http://localhost:3002 | |
| # Cloud: FIRECRAWL_API_BASE_URL=https://fastcrw.com | |
| # Self-hosted (bare metal, no Docker): FIRECRAWL_API_BASE_URL=http://localhost:3002 | |
| # Self-hosted (Docker): use a host/container-reachable URL, e.g. FIRECRAWL_API_BASE_URL=http://host.docker.internal:3002 |
- Clarify that FIRECRAWL_API_BASE_URL applies to server-side/proxy calls only - Add Docker note about using host.docker.internal instead of localhost - Add Docker host note in env.tpl template comments
Summary
localhost:3002) and cloud (fastcrw.com) usage viaFIRECRAWL_API_BASE_URLenv.tplfor discoverabilityDetails
CRW is fully compatible with the Firecrawl API, so users can switch to it by simply setting:
No code changes are needed — it works with the existing Firecrawl search provider selection.
Changes
README.md: Added CRW to the web search feature list and a new section with self-hosted/cloud setup instructionsenv.tpl: Added comments about CRW as an alternative forFIRECRAWL_API_BASE_URL