@@ -15,55 +15,65 @@ CLOUDFLARE_ACCESS_SUBDOMAIN="mlcommons"
1515CLOUDFLARE_ACCESS_LOGOUT_URL=" https://${CLOUDFLARE_ACCESS_SUBDOMAIN} .cloudflareaccess.com/cdn-cgi/access/logout"
1616
1717# Usage string
18- USAGE_STRING=" USAGE: bash [-d download-path] [-s] [-x] [-h] <URL>"
18+ USAGE_STRING=" USAGE: bash [-d download-path] [-s] [-t] [- x] [-h] <URL>"
1919
2020# Function to show help
2121show_help () {
2222 cat << EOF
23- MLCommons R2 Downloader - Download files from Cloudflare R2 buckets protected by Cloudflare Access
23+ MLCommons R2 Downloader - Download files from Cloudflare R2 buckets with or without Cloudflare Access authentication.
2424
2525$USAGE_STRING
2626
2727ARGUMENTS:
28- <URL> URL to the dataset metadata file (*.uri) on the R2 bucket
29- This file contains the base URL for the dataset files
28+ <URL> URL to the dataset metadata file (*.uri) on the R2 bucket.
29+ This file contains the base URL for the dataset files.
3030
3131OPTIONS:
32- -d download-path Directory where files will be downloaded
33- Defaults to the dataset name from the URL if not specified
34- -s Use service account credentials (requires CF_ACCESS_CLIENT_ID and
35- CF_ACCESS_CLIENT_SECRET environment variables)
36- -t Testing mode - forces cloudflared check/install even when using service account
37- -x Debug mode - shows parsed URL components and configuration then exits
38- -h Show this help message and exit
32+ -d download-path Directory where files will be downloaded.
33+ Defaults to the dataset name from the URL if not specified.
34+ -s Use service-account credentials for authentication (requires
35+ CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET environment variables).
36+ -t Testing mode: runs reduced, non-interactive checks and always verifies
37+ cloudflared is installed (useful for CI).
38+ -x Debug mode: prints parsed URL components and configuration then exits.
39+ -h Show this help message and exit.
3940
4041EXAMPLES:
41- # Download to current directory using dataset name
42- bash https://inference-private .mlcommons-storage.org/metadata/llama3 .uri
42+ # Download a public dataset to current directory using dataset name
43+ bash https://public-dataset .mlcommons-storage.org/metadata/dataset-name .uri
4344
4445 # Download to specific directory
45- bash -d ./ my-dataset https://inference-private .mlcommons-storage.org/metadata/llama3 .uri
46+ bash -d /path/to/ my-dataset https://public-dataset .mlcommons-storage.org/metadata/dataset-name .uri
4647
47- # Use service account authentication
48- bash -s https://inference- private.mlcommons-storage.org/metadata/llama3 .uri
48+ # Download a private dataset with interactive browser login
49+ bash https://private-dataset .mlcommons-storage.org/metadata/dataset-name .uri
4950
50- # Use service account with testing mode (checks cloudflared even with service account)
51- bash -s -t https://inference-private.mlcommons-storage.org/metadata/llama3.uri
51+ # Use a service account for non-interactive authentication
52+ # (make sure to set CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET environment variables)
53+ bash -s https://private-dataset.mlcommons-storage.org/metadata/dataset-name.uri
5254
53- # Debug mode to see how URL is parsed
54- bash -x https://inference-private.mlcommons-storage.org/metadata/llama3.uri
55+ # CI / automated test mode (service account + testing mode)
56+ bash -s -t https://private-dataset.mlcommons-storage.org/metadata/dataset-name.uri
57+
58+ # Debug mode to inspect URL parsing and configuration
59+ bash -x https://private-dataset.mlcommons-storage.org/metadata/dataset-name.uri
5560
5661REQUIREMENTS:
57- - cloudflared (will be auto-installed if missing)
5862 - wget
5963 - mktemp
6064 - md5sum/gmd5sum (for hash verification)
65+ - cloudflared (auto-installed only if authentication is required)
6166
6267AUTHENTICATION:
63- This script uses Cloudflare Access for authentication. On first run, it will:
64- 1. Open a browser window for you to authenticate
65- 2. Cache authentication tokens for future use
66- 3. Automatically re-authenticate when tokens expire
68+ The downloader automatically detects whether the requested dataset is protected by
69+ Cloudflare Access.
70+
71+ • Public dataset → No authentication needed, and cloudflared will be skipped.
72+ • Private dataset → The script will authenticate using one of the following methods:
73+ - Browser-based login (default)
74+ - Service account (-s) for headless/CI usage.
75+
76+ The script will automatically re-authenticate when authentication sessions approach expiration.
6777
6878EOF
6979}
0 commit comments