v3.7.3

3.7.3 (2024-01-30)

Bug Fixes

• enqueueLinks: filter out empty/nullish globs (#2286) (84319b3)
• pass on an invisible CF turnstile (#2277) (d8734e7), closes #2256

v3.7.2

3.7.2 (2024-01-09)

Bug Fixes

• RequestQueue: always clear locks when a request is reclaimed (#2263) (0fafe29), closes #2262

v3.7.1

3.7.1 (2024-01-02)

Bug Fixes

• ES2022 build compatibility and move to NodeNext for module (#2258) (7fe1e68), closes #2257

v3.7.0

3.7.0 (2023-12-21)

Bug Fixes

• retryOnBlocked doesn't override the blocked HTTP codes (#2243) (81672c3)
• browser-pool: respect user options before assigning fingerpints (#2190) (f050776), closes #2164
• filter out empty globs (#2205) (41322ab), closes #2200
• make CLI work on Windows too with --no-purge (#2244) (83f3179)
• make SessionPool queue up getSession calls to prevent overruns (#2239) (0f5665c), closes #1667
• MemoryStorage: lock request JSON file when reading to support multiple process crawling (#2215) (eb84ce9)

Features

• allow configuring crawler statistics (#2213) (9fd60e4), closes #1789
• check enqueue link strategy post redirect (#2238) (3c5f9d6), closes #2173
• log cause with retryOnBlocked (#2252) (e19a773), closes #2249
• robots.txt and sitemap.xml utils (#2214) (fdfec4f), closes #2187

v3.6.2

3.6.2 (2023-11-26)

Bug Fixes

• prevent race condition in KeyValueStore.getAutoSavedValue() (#2193) (e340e2b)

v3.6.1

3.6.1 (2023-11-15)

Bug Fixes

• ts: ignore import errors for got-scraping (012fc9e)
• ts: specify type explicitly for logger (aec3550)

Features

• puppeteer: enable new headless mode (#1910) (7fc999c)

v3.6.0

3.6.0 (2023-11-15)

Bug Fixes

• add skipNavigation option to enqueueLinks (#2153) (118515d)
• BrowserPool: ignore --no-sandbox flag for webkit launcher (#2148) (1eb2f08), closes #1797
• core: respect some advanced options for RequestList.open() + improve docs (#2158) (c5a1b07)
• declare missing dependency on got-scraping in the core package (cd2fd4d)
• provide more detailed error messages for browser launch errors (#2157) (f188ebe)
• retry incorrect Content-Type when response has blocked status code (#2176) (b54fb8b), closes #1994

Features

• core: add crawler.exportData() helper (#2166) (c8c09a5)
• got-scraping v4 (#2110) (2f05ed2)

v3.5.8

3.5.8 (2023-10-17)

Bug Fixes

• MemoryStorage: ignore invalid files for request queues (#2132) (fa58581), closes #1985
• refactor extractUrls to split the text line by line first (#2122) (7265cd7)

v3.5.7

3.5.7 (2023-10-05)

Bug Fixes

• add warning when we detect use of RL and RQ, but RQ is not provided explicitly (#2115) (6fb1c55), closes #1773
• ensure the status message cannot stuck the crawler (#2114) (9034f08)
• RQ request count is consistent after migration (#2116) (9ab8c18), closes #1855 #1855

v3.5.6

3.5.6 (2023-10-04)

Bug Fixes

• types: re-export RequestQueueOptions as an alias to RequestProviderOptions (#2109) (0900f76)

Features

• add incapsula iframe selector to the blocked list (#2111) (2b17d8a), closes apify/store-website-content-crawler#154