Skip to content

Commit 028026a

Browse files
committed
Update help output in light of no-auth support
1 parent 78d58f4 commit 028026a

1 file changed

Lines changed: 35 additions & 25 deletions

File tree

mlc-r2-downloader.sh

Lines changed: 35 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -15,55 +15,65 @@ CLOUDFLARE_ACCESS_SUBDOMAIN="mlcommons"
1515
CLOUDFLARE_ACCESS_LOGOUT_URL="https://${CLOUDFLARE_ACCESS_SUBDOMAIN}.cloudflareaccess.com/cdn-cgi/access/logout"
1616

1717
# Usage string
18-
USAGE_STRING="USAGE: bash [-d download-path] [-s] [-x] [-h] <URL>"
18+
USAGE_STRING="USAGE: bash [-d download-path] [-s] [-t] [-x] [-h] <URL>"
1919

2020
# Function to show help
2121
show_help() {
2222
cat << EOF
23-
MLCommons R2 Downloader - Download files from Cloudflare R2 buckets protected by Cloudflare Access
23+
MLCommons R2 Downloader - Download files from Cloudflare R2 buckets with or without Cloudflare Access authentication.
2424
2525
$USAGE_STRING
2626
2727
ARGUMENTS:
28-
<URL> URL to the dataset metadata file (*.uri) on the R2 bucket
29-
This file contains the base URL for the dataset files
28+
<URL> URL to the dataset metadata file (*.uri) on the R2 bucket.
29+
This file contains the base URL for the dataset files.
3030
3131
OPTIONS:
32-
-d download-path Directory where files will be downloaded
33-
Defaults to the dataset name from the URL if not specified
34-
-s Use service account credentials (requires CF_ACCESS_CLIENT_ID and
35-
CF_ACCESS_CLIENT_SECRET environment variables)
36-
-t Testing mode - forces cloudflared check/install even when using service account
37-
-x Debug mode - shows parsed URL components and configuration then exits
38-
-h Show this help message and exit
32+
-d download-path Directory where files will be downloaded.
33+
Defaults to the dataset name from the URL if not specified.
34+
-s Use service-account credentials for authentication (requires
35+
CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET environment variables).
36+
-t Testing mode: runs reduced, non-interactive checks and always verifies
37+
cloudflared is installed (useful for CI).
38+
-x Debug mode: prints parsed URL components and configuration then exits.
39+
-h Show this help message and exit.
3940
4041
EXAMPLES:
41-
# Download to current directory using dataset name
42-
bash https://inference-private.mlcommons-storage.org/metadata/llama3.uri
42+
# Download a public dataset to current directory using dataset name
43+
bash https://public-dataset.mlcommons-storage.org/metadata/dataset-name.uri
4344
4445
# Download to specific directory
45-
bash -d ./my-dataset https://inference-private.mlcommons-storage.org/metadata/llama3.uri
46+
bash -d /path/to/my-dataset https://public-dataset.mlcommons-storage.org/metadata/dataset-name.uri
4647
47-
# Use service account authentication
48-
bash -s https://inference-private.mlcommons-storage.org/metadata/llama3.uri
48+
# Download a private dataset with interactive browser login
49+
bash https://private-dataset.mlcommons-storage.org/metadata/dataset-name.uri
4950
50-
# Use service account with testing mode (checks cloudflared even with service account)
51-
bash -s -t https://inference-private.mlcommons-storage.org/metadata/llama3.uri
51+
# Use a service account for non-interactive authentication
52+
# (make sure to set CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET environment variables)
53+
bash -s https://private-dataset.mlcommons-storage.org/metadata/dataset-name.uri
5254
53-
# Debug mode to see how URL is parsed
54-
bash -x https://inference-private.mlcommons-storage.org/metadata/llama3.uri
55+
# CI / automated test mode (service account + testing mode)
56+
bash -s -t https://private-dataset.mlcommons-storage.org/metadata/dataset-name.uri
57+
58+
# Debug mode to inspect URL parsing and configuration
59+
bash -x https://private-dataset.mlcommons-storage.org/metadata/dataset-name.uri
5560
5661
REQUIREMENTS:
57-
- cloudflared (will be auto-installed if missing)
5862
- wget
5963
- mktemp
6064
- md5sum/gmd5sum (for hash verification)
65+
- cloudflared (auto-installed only if authentication is required)
6166
6267
AUTHENTICATION:
63-
This script uses Cloudflare Access for authentication. On first run, it will:
64-
1. Open a browser window for you to authenticate
65-
2. Cache authentication tokens for future use
66-
3. Automatically re-authenticate when tokens expire
68+
The downloader automatically detects whether the requested dataset is protected by
69+
Cloudflare Access.
70+
71+
• Public dataset → No authentication needed, and cloudflared will be skipped.
72+
• Private dataset → The script will authenticate using one of the following methods:
73+
- Browser-based login (default)
74+
- Service account (-s) for headless/CI usage.
75+
76+
The script will automatically re-authenticate when authentication sessions approach expiration.
6777
6878
EOF
6979
}

0 commit comments

Comments
 (0)