Skip to content

Refactor delete requests to use rate limiter #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sebtoombs wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Refactor delete requests to use rate limiter #1

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f09074b
Refactor delete requests to use rate limiter
sebtoombs Sep 13, 2022
963f7b3
Update index.js
sebtoombs Sep 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 28 additions & 33 deletions index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
const core = require("@actions/core");
const fetch = require("cross-fetch");
const {
RateLimiterMemory,
RateLimiterQueue,
} = require("rate-limiter-flexible");

// e.g. 1 request per 2 seconds
const limiterFlexible = new RateLimiterMemory({
points: 1,
duration: 2,
});

const limiterQueue = new RateLimiterQueue(limiterFlexible, {
maxQueueSize: 100,
});

const orgName = core.getInput("org_name", {
required: true,
Expand Down Expand Up @@ -46,13 +60,13 @@ const deleteOptions = {
};

/**
* getReleasess
* getReleases
*
* Fetch a list of releases and filter based on version number
*
* @returns Promise<number> - array of release ids to purge
*/
async function getReleasess() {
async function getReleases() {
const response = await fetch(appCenterUrl, getOptions);
if (!response.ok) {
throw new Error(
Expand Down Expand Up @@ -91,7 +105,7 @@ async function deleteRelease(releaseId) {

async function main() {
try {
const appReleases = await getReleasess();
const appReleases = await getReleases();

if (!appReleases.length) {
core.info(`No releases to purge`);
Expand All @@ -109,43 +123,24 @@ async function main() {
core.info(`All rows will be purged!`);
}

const releasesToPurge = [];
for (let i = 0; i < appReleases.length; i++) {
if (i >= toKeep) {
releasesToPurge.push(appReleases[i]);
}
}
// Only delete releases after the number to keep
const releasesToPurge = appReleases.slice(0, -toKeep);

@apkelly apkelly Sep 13, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this slice keep the head of the array or the tail? We want to keep the head (as these are the latest versions).

@sebtoombs sebtoombs Sep 13, 2022 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, this will remove the last toKeep items from the array - which I have just realised is incorrect. Pushed a change 🙏


core.debug(`Release IDs to purge: ${JSON.stringify(releasesToPurge)}`);

// Fire off all the delete requests (in any order the engine sees fit) and wait for all of them to either succeed or fail
// const deleteResult = await Promise.allSettled(
// releasesToPurge.map((releaseId) => {
// if (dryRun) {
// core.debug(`Dry run, not purging releaseId=${releaseId}`);
// return Promise.resolve(releaseId);
// }
// core.debug(`Purging releaseId=${releaseId}`);
// return deleteRelease(releaseId);
// })
// );

// Run all delete requests in sequence (slooooooow)
const deleteResult = [];
for (const releaseId of releasesToPurge) {
try {
// Run all delete requests in parallel (fast) - but limited by the rate limiter settings
const deleteResult = await Promise.allSettled(
releasesToPurge.map(async (releaseId) => {
await limiterQueue.removeTokens(1);

if (dryRun) {
core.debug(`Dry run, not purging releaseId=${releaseId}`);
deleteResult.push({ status: "fulfilled", value: releaseId });
continue;
return releaseId;
}
core.debug(`Purging releaseId=${releaseId}`);
const result = await deleteRelease(releaseId);
deleteResult.push({ status: "fulfilled", value: result });
} catch (e) {
deleteResult.push({ status: "rejected", reason: e });
}
}
return deleteRelease(releaseId);
})
);

// Grab the IDs of the releases that succeeded
const succeededDeletedIds = deleteResult
Expand Down
3 changes: 2 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
"homepage": "https://github.com/mx51/AppCenter-Purge-Github-Action#readme",
"dependencies": {
"@actions/core": "^1.9.1",
"cross-fetch": "^3.1.5"
"cross-fetch": "^3.1.5",
"rate-limiter-flexible": "^2.3.10"
}
}