Downloading Llama2 on SUSE Linux fails with unsupported OS error

[psyhtest]

Attempting to download either Llama2 Training Data (Scrolls Gov Report 8k):

bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
https://llama2.mlcommons-storage.org/metadata/scrolls-gov-report-8k.uri

or Training Model (Llama 2 70B Parameter Fused QKV MLPerf Model):

bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
https://llama2.mlcommons-storage.org/metadata/llama-2-70b-fused-qkv-mlperf.uri

results in:

Checking if authentication is required...
Authentication required
Checking dependencies...
cloudflared not found. Attempting to install...
Installing cloudflared...
Error: Unsupported operating system

System details

anton.u5bz@login41:~> cat /etc/*release
NAME="SLES"
VERSION="15-SP6"
VERSION_ID="15.6"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp6"
DOCUMENTATION_URL="https://documentation.suse.com/"
|
anton.u5bz@login41:~> uname -a
Linux login41 6.4.0-150600.23.53_15.0.11-cray_shasta_c_64k #1 SMP Tue Jul 15 19:45:15 UTC 2025 (80d95d6) aarch64 aarch64 aarch64 GNU/Linux

[psyhtest]

Any reason why Llama2 downloads are not listed among the other Training downloads?

By the way, if I try to download e.g. the Llama3.1-405B validation dataset on the same system, I encounter no error.

Details

anton.u5bz@login41:~> bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://training.mlcommons-storage.org/metadata/c4-validation-dataset-zipped.uri
Checking if authentication is required...
No authentication required
Downloading dataset information...
Downloading checksums file...
Detected single-file dataset. Will download to current directory.
Preparing download...
Dataset: c4-validation-dataset-zipped
Download directory: .
Preparing file list for download...
Starting download of dataset files...
--2025-09-22 23:50:17--  https://training.mlcommons-storage.org/common/datasets/c4/original/c4-validation-91205-samples.en.json.gz
Resolving training.mlcommons-storage.org (training.mlcommons-storage.org)... 172.67.74.32, 104.26.1.198, 104.26.0.198, ...
Connecting to training.mlcommons-storage.org (training.mlcommons-storage.org)|172.67.74.32|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 81200017 (77M) [application/gzip]
Saving to: ‘./c4-validation-91205-samples.en.json.gz’
|
c4-validation-91205-samples.en.json.gz              100%[===================================================================================================================>]  77.44M  84.7MB/s    in 0.9s
|
2025-09-22 23:50:18 (84.7 MB/s) - ‘./c4-validation-91205-samples.en.json.gz’ saved [81200017/81200017]
|
FINISHED --2025-09-22 23:50:18--
Total wall clock time: 1.5s
Downloaded: 1 files, 77M in 0.9s (84.7 MB/s)
Download completed successfully!
Verifying file integrity with MD5 checksums...
c4-validation-91205-samples.en.json.gz: OK
Checksum verification completed successfully!
All files have been downloaded and verified in: .

[nathanwasson]

@psyhtest Could you provide the output of the following:

echo $OSTYPE

[nathanwasson]

Any reason why Llama2 downloads are not listed among the other Training downloads?

The downloads pages are tied to each bucket, and the Llama 2 models and associated data are stored in separate buckets with restricted access, requiring that those seeking to download the data be MLC Members and have signed the Llama 2 confidentiality agreement. The Llama 2 data is not widely available to the public the way the data listed on the primary Training downloads page is.

[nathanwasson]

This is very strange. I'm unable to reproduce the reported behavior in the official SUSE Linux Enterprise Server 15 SP6 container image. I successfully downloaded Scrolls Gov Report 8k within the container. I suppose $OSTYPE could be set differently on a bare metal installation.

[nathanwasson]

Okay, I found the problem. I installed SUSE Linux Enterprise Server 15 SP6 in a VM, and it sets $OSTYPE to linux instead of linux-gnu. I'm putting together a fix for the script.

[nathanwasson]

Alright, I switched the OS detection logic to use uname instead of $OSTYPE, which should provide broader compatibility beyond even just SUSE; it should also handle Alpine Linux and other musl-based distros. The uname method passed all tests and functioned properly within my SUSE VM, so I went ahead and merged it into main.

@psyhtest please try the download commands again and let me know how it goes.

[psyhtest]

Thank you very much @nathanwasson! Yes, OSTYPE=linux.

I get further but it attempts to install cloudflared with sudo which I don't and can't have on this machine. Can it be installed locally perhaps into $HOME/.local/bin instead?

bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://llama2.mlcommons-storage.org/metadata/scrolls-gov-report-8k.uri

Checking if authentication is required...
Authentication required
Checking dependencies...
cloudflared not found. Attempting to install...
Installing cloudflared...
Downloading cloudflared...
|
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
|
    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.
|
For security reasons, the password you type will not be visible.
|
[sudo] password for root:
sudo: PAM authentication error: User not known to the underlying authentication module
sudo: a password is required
Error: Failed to install cloudflared. Try running with sudo.

[psyhtest]

By the way, the Llama2 reference README still mentions rclone in the download section.

[psyhtest]

Success! Here's what I did:

1. Downloaded the latest cloudflared-linux-arm64 binary from here.
2. Copied it into ~/.local/bin.
3. Created a symlink there: ln -s cloudfrared-linux-arm64 cloudflared
4. Run the data download command.

cloudflared --version

cloudflared version 2025.9.1 (built 2025-09-22-13:28 UTC)

bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://llama2.mlcommons-storage.org/metadata/scrolls-gov-report-8k.uri

Checking if authentication is required...
Authentication required
Checking dependencies...
Logging in to Cloudflare Access at https://llama2.mlcommons-storage.org/metadata/scrolls-gov-report-8k.uri...
Please open the following URL and log in with your Cloudflare account:
|
https://llama2.mlcommons-storage.org/cdn-cgi/access/<...>
|
Leave cloudflared running to download the token automatically.
|
Authentication successful!
Checking token expiration time...
Cloudflare Access token will expire in 13 days, 16 hours
Downloading dataset information...
|
Downloading checksums file...
Detected multi-file dataset (2 files). Will download to subdirectory.
Preparing download...
Dataset: scrolls-gov-report-8k
Download directory: scrolls_gov_report_8k
Creating download directory: scrolls_gov_report_8k
Preparing file list for download...
Starting download of dataset files...
--2025-09-23 07:10:52--  https://llama2.mlcommons-storage.org/data/training/scrolls_gov_report_8k/validation-00000-of-00001.parquet
Resolving llama2.mlcommons-storage.org (llama2.mlcommons-storage.org)... 104.26.0.198, 172.67.74.32, 104.26.1.198, ...
Connecting to llama2.mlcommons-storage.org (llama2.mlcommons-storage.org)|104.26.0.198|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2685523 (2.6M) [application/octet-stream]
Saving to: ‘scrolls_gov_report_8k/validation-00000-of-00001.parquet’
|
validation-00000-of-00001.parquet                   100%[===================================================================================================================>]   2.56M  --.-KB/s    in 0.07s
|
2025-09-23 07:10:52 (36.3 MB/s) - ‘scrolls_gov_report_8k/validation-00000-of-00001.parquet’ saved [2685523/2685523]
|
--2025-09-23 07:10:52--  https://llama2.mlcommons-storage.org/data/training/scrolls_gov_report_8k/train-00000-of-00001.parquet
Reusing existing connection to llama2.mlcommons-storage.org:443.
HTTP request sent, awaiting response... 200 OK
Length: 108961020 (104M) [application/octet-stream]
Saving to: ‘scrolls_gov_report_8k/train-00000-of-00001.parquet’
|
train-00000-of-00001.parquet                        100%[===================================================================================================================>] 103.91M  21.8MB/s    in 4.7s
|
2025-09-23 07:10:58 (22.2 MB/s) - ‘scrolls_gov_report_8k/train-00000-of-00001.parquet’ saved [108961020/108961020]
|
FINISHED --2025-09-23 07:10:58--
Total wall clock time: 5.7s
Downloaded: 2 files, 106M in 4.8s (22.4 MB/s)
Download completed successfully!
Verifying file integrity with MD5 checksums...
validation-00000-of-00001.parquet: OK
train-00000-of-00001.parquet: OK
Checksum verification completed successfully!
All files have been downloaded and verified in: scrolls_gov_report_8k

Onto the model download!

[nathanwasson]

Thanks for working through this. Since, as you point out, some people don't/can't have sudo on their machines for the cloudflared install, I've added a container image to the repo that comes with all the dependencies and the latest version of the script. Hopefully this option helps some folks get around dependency issues in locked-down environments.

[nathanwasson]

Closing this issue as resolved with #20 and #21.