Skip to content

how to handle timeout error #2621

Description

@zy783282949

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

I am using CheerioCrawler. When I run the crawler.run() method for a while, an error occurs:
Handling request failure of https://xxx(ZNGIMiMhlwKFihT) timed out after 300 seconds.
I hope to drop this data and continue execution instead of exiting from the main process.

Code sample

The pseudo code is as follows: Two functions will be executed simultaneously in the main function, one is the enqueueProcess function, which continuously fetches data from redis and calls the crawler.addRequest() method to add it to the queue, and the other is the crawler.run() method

async function enqueueProcess(){
while (true){
crawler.addRequest("xxx")
}
}
async function main(){
const crawler = new CheerioCrawler();
enququeProcess();
crawler.run();
}

Package version

├── crawlee@3.10.0 ├── got@14.3.0 ├── http-proxy-agent@7.0.2 ├── https-proxy-agent@7.0.4 └── tunnel@0.0.6

Node.js version

v20.16.0

Operating system

centos8

Apify platform

  • Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions