You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current implementation (added in feat: add support for sameDomainDelay #2003) essentially holds requests to recently accessed domains in memory and re-enqueues them after a given timeout
⚠️ this may result in lots of unnecessary request queue writes
⚠️ there may be unexpected results if this is used in conjunction with request locking
but resolving this for multiple request queue consumers would be quite an endeavor
we cannot simply use maxRequestsPerMinute - we don't know the request URL (and thus the domain) before receiving it from the queue
Regardless of the resolution, the "final" version should be ported over to crawlee-python.
sameDomainDelay#2003) essentially holds requests to recently accessed domains in memory and re-enqueues them after a given timeoutmaxRequestsPerMinute- we don't know the request URL (and thus the domain) before receiving it from the queueRegardless of the resolution, the "final" version should be ported over to crawlee-python.