Skip to content

Commit 518c8ba

Browse files
committed
Add batch processing support for parallel repository execution in gh repo-stats
1 parent 7417918 commit 518c8ba

2 files changed

Lines changed: 776 additions & 51 deletions

File tree

README.md

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@ Options:
4444
potential naming conflicts if those orgs are to be merged during migration
4545
-y, --token-type : Type of Personal Access, can either be "user" or "app"
4646
Default: user
47+
-b, --batch : Number of repositories to process in parallel
48+
Creates temp CSV files, then merges and sorts by repo name
49+
Default: 0 (disabled, sequential processing)
4750
4851
Description:
4952
Scans an organization or list of organizations for all repositories and gathers size statistics for each repository
@@ -90,7 +93,7 @@ tinyfists,actions-experiments,false,2023-03-10T16:15:27Z,2022-10-28T19:38:34Z,ab
9093
- `Latest_Commit_Branch`: Branch name where the most recent commit was made
9194
- `Default_Branch`: Default branch of the repository (e.g., main, master), or "No default branch" if none exists
9295
- `isFork`: Whether the repository is a fork
93-
- `isArchive`: Whether the repository is archived
96+
- `isArchived`: Whether the repository is archived
9497
- `Repo_Size_mb`: Size of the repository in megabytes
9598
- `Record_Count`: Number of database records this repository represents
9699
- `Collaborator_Count`: Number of users who have contributed to this repository
@@ -115,6 +118,43 @@ tinyfists,actions-experiments,false,2023-03-10T16:15:27Z,2022-10-28T19:38:34Z,ab
115118
- 1.5 GB or larger size on disk
116119
- `Created`: Date/time when the repository was created
117120
121+
## Batch Processing (Parallel Execution)
122+
123+
For organizations with many repositories, you can use the `-b` or `--batch` option to process repositories in parallel, significantly reducing the total execution time.
124+
125+
### How It Works
126+
127+
1. **Fetches all repository names** first with a lightweight query
128+
2. **Distributes repositories** across the specified number of parallel workers
129+
3. **Creates temporary CSV files** in `./temp_files/` directory (one per worker)
130+
4. **Merges and sorts** all results alphabetically by repository name into the final output file
131+
5. **Cleans up** temporary files automatically after completion (or on interruption)
132+
133+
### Usage Examples
134+
135+
```shell
136+
# Process with 4 parallel workers
137+
gh repo-stats -o my-org-name --batch 4
138+
139+
# Combine with other options
140+
gh repo-stats -o my-org-name -b 8 -c fields-config.csv
141+
```
142+
143+
### Progress Tracking
144+
145+
When batch mode is enabled, you'll see a real-time progress bar:
146+
147+
```
148+
🚀 Progress: [████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 30% | ✅ 30/100 repos | ⚡ 4 active | 🔄 Processing...
149+
```
150+
151+
### Notes
152+
153+
- The output is automatically sorted alphabetically by repository name
154+
- Temporary files are cleaned up automatically on completion or interruption (Ctrl+C)
155+
- If interrupted, the script will terminate all background workers and clean up temp files
156+
- Recommended batch size: 4-8 workers (higher values may hit API rate limits faster)
157+
118158
## Field Configuration
119159

120160
You can customize which fields are included in the output by using a field configuration CSV file with the `-c` or `--config` option. This is useful when you only need specific fields and want to reduce the output size.

0 commit comments

Comments
 (0)