Which package is this bug report for? If unsure which one to select, leave blank
None
Issue description
I am using CheerioCrawler. When I run the crawler.run() method for a while, an error occurs:
Handling request failure of https://xxx(ZNGIMiMhlwKFihT) timed out after 300 seconds.
I hope to drop this data and continue execution instead of exiting from the main process.
Code sample
The pseudo code is as follows: Two functions will be executed simultaneously in the main function, one is the enqueueProcess function, which continuously fetches data from redis and calls the crawler.addRequest() method to add it to the queue, and the other is the crawler.run() method
async function enqueueProcess(){
while (true){
crawler.addRequest("xxx")
}
}
async function main(){
const crawler = new CheerioCrawler();
enququeProcess();
crawler.run();
}
Package version
├── crawlee@3.10.0 ├── got@14.3.0 ├── http-proxy-agent@7.0.2 ├── https-proxy-agent@7.0.4 └── tunnel@0.0.6
Node.js version
v20.16.0
Operating system
centos8
Apify platform
I have tested this on the next release
No response
Other context
No response
Which package is this bug report for? If unsure which one to select, leave blank
None
Issue description
I am using CheerioCrawler. When I run the crawler.run() method for a while, an error occurs:
Handling request failure of https://xxx(ZNGIMiMhlwKFihT) timed out after 300 seconds.
I hope to drop this data and continue execution instead of exiting from the main process.
Code sample
Package version
├── crawlee@3.10.0 ├── got@14.3.0 ├── http-proxy-agent@7.0.2 ├── https-proxy-agent@7.0.4 └── tunnel@0.0.6
Node.js version
v20.16.0
Operating system
centos8
Apify platform
I have tested this on the
nextreleaseNo response
Other context
No response