key value storage file operation error causes whole application failure

[RAHUL1115]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/cheerio (CheerioCrawler)

Issue description

the issue in not that easy to reproduce. and i am getting the error on version 3 on version 2 things were running very smoothly.

log file content:

Error: EPERM: operation not permitted, unlink 'C:\Users\name\Downloads\scraper-v7\_crawler_storage\key_value_stores\default\SDK_CRAWLER_STATISTICS_50.json'
2023-01-13T06:48:55.150Z error: getLinksToScrape.service.js : uncaughtException: EPERM: operation not permitted, unlink 'C:\Users\name\Downloads\scraper-v7\_crawler_storage\key_value_stores\default\SDK_CRAWLER_STATISTICS_50.json'
Error: EPERM: operation not permitted, unlink 'C:\Users\name\Downloads\scraper-v7\_crawler_storage\key_value_stores\default\SDK_CRAWLER_STATISTICS_50.json'
2023-01-13T06:48:55.155Z error: scrapper-script.js : uncaughtException: EPERM: operation not permitted, unlink 'C:\Users\name\Downloads\scraper-v7\_crawler_storage\key_value_stores\default\SDK_CRAWLER_STATISTICS_50.json'
Error: EPERM: operation not permitted, unlink 'C:\Users\name\Downloads\scraper-v7\_crawler_storage\key_value_stores\default\SDK_CRAWLER_STATISTICS_50.json'
2023-01-13T06:48:55.160Z error: web.js.js : uncaughtException: EPERM: operation not permitted, unlink 'C:\Users\name\Downloads\scraper-v7\_crawler_storage\key_value_stores\default\SDK_CRAWLER_STATISTICS_50.json'
Error: EPERM: operation not permitted, unlink 'C:\Users\name\Downloads\scraper-v7\_crawler_storage\key_value_stores\default\SDK_CRAWLER_STATISTICS_50.json'

Code sample

import { RequestQueue, CheerioCrawler } from 'crawlee';

let urls = [['http://url1.com/1.html'],['http://url1.com/2.html'],['http://url2.com/1.html'],['http://url1.com/2.html']] //array of urls

for (let index = 0; index < urls.length; index++) { // limit to only 10 rquest in queue

    const crawler = new CheerioCrawler({
        async requestHandler({ $, request }) {
            const title = $('title').text();
            console.log(`The title of "${request.url}" is: ${title}.`);
        }
    })

    await crawler.run(urls[index]);
}

Package version

3.1.4

Node.js version

16

Operating system

windows

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

3.1.5.beta8

Other context

No response

[LeMoussel]

On my windows system, Crawlee V 3.1.4, Node JS V 16.17.1
No Errors with this code. It seems to be a permissions Windows issue on your system.

import { CheerioCrawler } from 'crawlee';

let urls = [
    ['http://example.com/1.html'],
    ['http://example.com/2.html'],
    ['http://example.com/3.html'],
    ['http://example.com/4.html'],
] //array of urls

for (const element of urls) { // limit to only 10 rquest in queue
    const crawler = new CheerioCrawler({
        async requestHandler({ $, request }) {
            const title = $('title').text();
            console.log(`The title of "${request.url}" is: ${title}.`);
        }
    })

    await crawler.run(element);
}

Why do you loop on the urls?
it's not necessary. You can just do

import { CheerioCrawler } from 'crawlee';

let urls = [
    'http://example.com/1.html',
    'http://example.com/2.html',
    'http://example.com/3.html',
    'http://example.com/4.html',
] //array of urls

const crawler = new CheerioCrawler({
    async requestHandler({ $, request }) {
        const title = $('title').text();
        console.log(`The title of "${request.url}" is: ${title}.`);
    }
})

await crawler.run(urls);

And limit to 10 requests per one crawl with maxRequestsPerCrawl option.