Releases: apify/crawlee
Releases · apify/crawlee
Release list
v0.20.2
v0.20.0
- BREAKING:
Apify.utils.requestAsBrowser()no longer aborts request on status code 406
or when other thantext/htmltype is received. Useoptions.abortFunctionif you want to
retain this functionality. - BREAKING: Added
useInsecureHttpParseroption toApify.utils.requestAsBrowser()which
istrueby default and forces the function to use a HTTP parser that is less strict than
default Node 12 parser, but also less secure. It is needed to be able to bypass certain
anti-scraping walls and fetch websites that do not comply with HTTP spec. - BREAKING:
RequestListnow removes all the elements from thesourcesarray on
initialization. If you need to use the sources somewhere else, make a copy. This change
was added as one of several measures to improve memory management ofRequestList
in scenarios with very large amount ofRequestinstances. - DEPRECATED:
RequestListOptions.persistSourcesKeyis now deprecated. Please use
RequestListOptions.persistRequestsKey. RequestListOptions.sourcescan now be an array ofstringURLs as well.- Added
sourcesFunctiontoRequestListOptions. It enables dynamic fetching of sources
and will only be called if persistedRequestswere not retrieved from key-value store.
Use it to reduce memory spikes and also to make sure that your sources are not re-created
on actor restarts. - Updated
stealthhiding ofwebdriverto avoid recent detections. Apify.utils.lognow points to an updated logger instance which prints colored logs (in TTY)
and supports overriding with custom loggers.- Improved
Apify.launchPuppeteer()code to prevent triggering bugs in Puppeteer by passing
more than required options topuppeteer.launch(). - Documented
BasicCrawler.autoscaledPoolproperty, and addedCheerioCrawler.autoscaledPool
andPuppeteerCrawler.autoscaledPoolproperties. SessionPoolnow persists state onteardown. Before, it only persisted state every minute.
This ensures that after a crawler finishes, the state is correctly persisted.- Added TypeScript typings and typedef documentation for all entities used throughout SDK.
- Upgraded
proxy-chainNPM package from 0.2.7 to 0.4.1 and many other dependencies - Removed all usage of the now deprecated
requestpackage.
v0.19.1
v0.19.0
- BREAKING:
APIFY_LOCAL_EMULATION_DIRenv var is no longer supported (deprecated on 2018-09-11).
UseAPIFY_LOCAL_STORAGE_DIRinstead. SessionPoolAPI updates and fixes. The API is no longer considered experimental.- Logging of system info moved from
requiretime toApify.main()invocation. - Use native
RegExpinstead ofxregexpfor unicode property escapes.
v0.18.1
v0.18.0
- BREAKING
CheerioCrawlerignores ssl errors by default -options.ignoreSslErrors: true. - Add
SessionPoolimplemenation toCheerioCrawler. - Add
SessionPoolimplementation toPuppeteerPoolandPupeteerCrawler. - Fix
Requestconstructor not making a copy of objects such asuserDataandheaders. - Fix
descoption not being applied in localdataset.getData().
v0.17.0
- BREAKING: Node 8 and 9 are no longer supported. Please use Node 10.17.0 or higher.
- DEPRECATED:
Apify.callTask()bodyandcontentTypeoptions are now deprecated.
Useinputinstead. It must be ofcontent-type: application/json. - Add default
SessionPoolimplementation toBasicCrawler. - Add the ability to create ad-hoc webhooks via
Apify.call()andApify.callTask(). - Add an example of form filling with
Puppeteer. - Add
countryoption toApify.getApifyProxyUrl(). - Add
Apify.utils.puppeteer.saveSnapshot()helper to quickly save HTML and screenshot of a page. - Add the ability to pass
gotsupported options torequestOptionsinCheerioCrawler
thus supporting things such ascookieJaragain. - Switch Puppeteer to web socket again due to suspected
pipeerrors. - Fix an issue where some encodings were not correctly parsed in
CheerioCrawler. - Fix parsing bad Content-Type headers for
CheerioCrawler. - Fix custom headers not being correctly applied in
Apify.utils.requestAsBrowser(). - Fix dataset limits not being correctly applied.
- Fix a race condition in
RequestQueueLocal. - Fix
RequestListpersistence of downloaded sources in key-value store. - Fix
Apify.utils.puppeteer.blockRequests()always including default patterns. - Fix inconsistent behavior of
Apify.utils.puppeteer.infiniteScroll()on some websites. - Fix retry histogram statistics sometimes showing invalid counts.
- Added regexps for Youtube videos (
YOUTUBE_REGEX,YOUTUBE_REGEX_GLOBAL) toutils.social - Added documentation for option
jsonin handlePageFunction ofCheerioCrawler
v0.16.1
- Add
useIncognitoPagesoption toPuppeteerPoolto enable opening new pages in incognito
browser contexts. This is useful to keep cookies and cache unique for each page. - Added options to load every content type in CheerioCrawler.
There are new optionsbodyandcontentTypeinhandlePageFunctionfor this purposes. - DEPRECATED: CheerioCrawler
htmloption inhandlePageFunctionwas replaced withbodyoption.