Meta-Llama-3.1-8B-Instruct_lora_2c.log

W1020 09:25:07.852000 2402310 site-packages/torch/distributed/run.py:811] 
W1020 09:25:07.852000 2402310 site-packages/torch/distributed/run.py:811] *****************************************
W1020 09:25:07.852000 2402310 site-packages/torch/distributed/run.py:811] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W1020 09:25:07.852000 2402310 site-packages/torch/distributed/run.py:811] *****************************************
Running LoRAFinetuneRecipeDistributed with resolved config:

batch_size: 2
batch_size_val: 2
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /tmp/Meta-Llama-3.1-8B-Instruct/
  checkpoint_files:
  - model-00001-of-00004.safetensors
  - model-00002-of-00004.safetensors
  - model-00003-of-00004.safetensors
  - model-00004-of-00004.safetensors
  model_type: LLAMA3
  output_dir: /tmp/torchtune/llama3_1_8B/lora
  recipe_checkpoint: null
clip_grad_norm: null
compile: false
dataset:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
  packed: true
  split: train[:95%]
dataset_val:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
  split: train[95%:]
device: xpu
dtype: bf16
enable_activation_checkpointing: false
enable_activation_offloading: false
epochs: 1
gradient_accumulation_steps: 8
log_every_n_steps: 1
log_level: INFO
log_peak_memory_stats: true
loss:
  _component_: torchtune.modules.loss.LinearCrossEntropyLoss
lr_scheduler:
  _component_: torchtune.training.lr_schedulers.get_cosine_schedule_with_warmup
  num_warmup_steps: 100
max_steps_per_epoch: 10
metric_logger:
  _component_: torchtune.training.metric_logging.DiskLogger
  log_dir: /tmp/torchtune/llama3_1_8B/lora/logs
model:
  _component_: torchtune.models.llama3_1.lora_llama3_1_8b
  apply_lora_to_mlp: true
  apply_lora_to_output: false
  lora_alpha: 16
  lora_attn_modules:
  - q_proj
  - v_proj
  - output_proj
  lora_dropout: 0.0
  lora_rank: 8
optimizer:
  _component_: torch.optim.AdamW
  fused: true
  lr: 0.0003
  weight_decay: 0.01
output_dir: /tmp/torchtune/llama3_1_8B/lora
profiler:
  _component_: torchtune.training.setup_torch_profiler
  active_steps: 2
  cpu: true
  cuda: true
  enabled: false
  num_cycles: 1
  output_dir: /tmp/torchtune/llama3_1_8B/lora/profiling_outputs
  profile_memory: false
  record_shapes: true
  wait_steps: 5
  warmup_steps: 3
  with_flops: false
  with_stack: false
resume_from_checkpoint: false
run_val_every_n_steps: null
save_adapter_weights_only: false
seed: 123
shuffle: true
tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  max_seq_len: 512
  path: /tmp/Meta-Llama-3.1-8B-Instruct/original/tokenizer.model

Running LoRAFinetuneRecipeDistributed with resolved config:

batch_size: 2
batch_size_val: 2
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /tmp/Meta-Llama-3.1-8B-Instruct/
  checkpoint_files:
  - model-00001-of-00004.safetensors
  - model-00002-of-00004.safetensors
  - model-00003-of-00004.safetensors
  - model-00004-of-00004.safetensors
  model_type: LLAMA3
  output_dir: /tmp/torchtune/llama3_1_8B/lora
  recipe_checkpoint: null
clip_grad_norm: null
compile: false
dataset:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
  packed: true
  split: train[:95%]
dataset_val:
  _component_: torchtune.datasets.alpaca_cleaned_dataset
  split: train[95%:]
device: xpu
dtype: bf16
enable_activation_checkpointing: false
enable_activation_offloading: false
epochs: 1
gradient_accumulation_steps: 8
log_every_n_steps: 1
log_level: INFO
log_peak_memory_stats: true
loss:
  _component_: torchtune.modules.loss.LinearCrossEntropyLoss
lr_scheduler:
  _component_: torchtune.training.lr_schedulers.get_cosine_schedule_with_warmup
  num_warmup_steps: 100
max_steps_per_epoch: 10
metric_logger:
  _component_: torchtune.training.metric_logging.DiskLogger
  log_dir: /tmp/torchtune/llama3_1_8B/lora/logs
model:
  _component_: torchtune.models.llama3_1.lora_llama3_1_8b
  apply_lora_to_mlp: true
  apply_lora_to_output: false
  lora_alpha: 16
  lora_attn_modules:
  - q_proj
  - v_proj
  - output_proj
  lora_dropout: 0.0
  lora_rank: 8
optimizer:
  _component_: torch.optim.AdamW
  fused: true
  lr: 0.0003
  weight_decay: 0.01
output_dir: /tmp/torchtune/llama3_1_8B/lora
profiler:
  _component_: torchtune.training.setup_torch_profiler
  active_steps: 2
  cpu: true
  cuda: true
  enabled: false
  num_cycles: 1
  output_dir: /tmp/torchtune/llama3_1_8B/lora/profiling_outputs
  profile_memory: false
  record_shapes: true
  wait_steps: 5
  warmup_steps: 3
  with_flops: false
  with_stack: false
resume_from_checkpoint: false
run_val_every_n_steps: null
save_adapter_weights_only: false
seed: 123
shuffle: true
tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  max_seq_len: 512
  path: /tmp/Meta-Llama-3.1-8B-Instruct/original/tokenizer.model

[Gloo] Rank 0 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
[Gloo] Rank 1 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
Writing logs to /tmp/torchtune/llama3_1_8B/lora/logs/log_1760952310.txt
FSDP is enabled. Instantiating model and loading checkpoint on Rank 0 ...
2025:10:20-09:25:11:2402384 |CCL_WARN| did not find MPI-launcher specific variables, switch to ATL/OFI, to force enable ATL/MPI set CCL_ATL_TRANSPORT=mpi
2025:10:20-09:25:11:2402384 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
2025:10:20-09:25:11:2402385 |CCL_WARN| did not find MPI-launcher specific variables, switch to ATL/OFI, to force enable ATL/MPI set CCL_ATL_TRANSPORT=mpi
2025:10:20-09:25:11:2402385 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
[rank1]:W1020 09:25:12.392000 2402385 site-packages/torch/distributed/tensor/_random.py:209] DTensor is synchronizing RNG states of every rank with the state from rank 0. This behavior is deprecated. Please call `torch.manual_seed()` on every rank that participates in SPMD DTensor Operations with the same seed. If using Pipeline Parallelism, each pipeling state would use a different seed, but all ranks belonging to one pipeline stage would use the same seed.
/home/jenkins/xiangdong/torchtune/recipes/lora_finetune_distributed.py:535: FutureWarning: lora_attn_modules is deprecated for validate_missing_and_unexpected_for_lora and will be removed in future versions. Please use state_dict_keys instead.
  validate_missing_and_unexpected_for_lora(
/home/jenkins/xiangdong/torchtune/torchtune/utils/_logging.py:143: FutureWarning: apply_lora_to_mlp is deprecated for validate_missing_and_unexpected_for_lora and will be removed in future versions. Please use state_dict_keys instead.
  return obj(*args, **kwargs)
/home/jenkins/xiangdong/torchtune/torchtune/utils/_logging.py:143: FutureWarning: apply_lora_to_output is deprecated for validate_missing_and_unexpected_for_lora and will be removed in future versions. Please use state_dict_keys instead.
  return obj(*args, **kwargs)
Instantiating model and loading checkpoint took 4.04 secs
Memory stats after model init:
	XPU peak memory active: 8.49 GiB
	XPU peak memory alloc: 8.49 GiB
	XPU peak memory reserved: 8.64 GiB
Optimizer is initialized.
Loss is initialized.

Packing dataset:   0%|          | 0/49172 [00:00<?, ?it/s]
Packing dataset:   1%|          | 329/49172 [00:00<00:14, 3287.87it/s]
Packing dataset:   1%|▏         | 658/49172 [00:00<00:14, 3274.56it/s]
Packing dataset:   2%|▏         | 986/49172 [00:00<00:15, 3150.46it/s]
Packing dataset:   3%|▎         | 1330/49172 [00:00<00:14, 3258.45it/s]
Packing dataset:   3%|▎         | 1657/49172 [00:00<00:15, 3158.39it/s]
Packing dataset:   4%|▍         | 1974/49172 [00:00<00:15, 3133.30it/s]
Packing dataset:   5%|▍         | 2296/49172 [00:00<00:14, 3159.52it/s]
Packing dataset:   5%|▌         | 2635/49172 [00:00<00:14, 3231.35it/s]
Packing dataset:   6%|▌         | 2972/49172 [00:00<00:14, 3273.23it/s]
Packing dataset:   7%|▋         | 3301/49172 [00:01<00:13, 3277.97it/s]
Packing dataset:   7%|▋         | 3630/49172 [00:01<00:14, 3248.00it/s]
Packing dataset:   8%|▊         | 3959/49172 [00:01<00:13, 3259.58it/s]
Packing dataset:   9%|▊         | 4286/49172 [00:01<00:13, 3246.67it/s]
Packing dataset:   9%|▉         | 4611/49172 [00:01<00:14, 3180.97it/s]
Packing dataset:  10%|█         | 4930/49172 [00:01<00:13, 3171.57it/s]
Packing dataset:  11%|█         | 5248/49172 [00:01<00:13, 3158.04it/s]
Packing dataset:  11%|█▏        | 5582/49172 [00:01<00:13, 3209.68it/s]
Packing dataset:  12%|█▏        | 5906/49172 [00:01<00:13, 3213.61it/s]
Packing dataset:  13%|█▎        | 6234/49172 [00:01<00:13, 3233.33it/s]
Packing dataset:  13%|█▎        | 6558/49172 [00:02<00:13, 3183.88it/s]
Packing dataset:  14%|█▍        | 6879/49172 [00:02<00:13, 3191.24it/s]
Packing dataset:  15%|█▍        | 7199/49172 [00:02<00:13, 3162.32it/s]
Packing dataset:  15%|█▌        | 7516/49172 [00:02<00:13, 3114.64it/s]
Packing dataset:  16%|█▌        | 7840/49172 [00:02<00:13, 3146.19it/s]
Packing dataset:  17%|█▋        | 8155/49172 [00:02<00:13, 3139.37it/s]
Packing dataset:  17%|█▋        | 8470/49172 [00:02<00:13, 3105.89it/s]
Packing dataset:  18%|█▊        | 8798/49172 [00:02<00:12, 3155.85it/s]
Packing dataset:  19%|█▊        | 9117/49172 [00:02<00:12, 3162.36it/s]
Packing dataset:  19%|█▉        | 9443/49172 [00:02<00:12, 3189.27it/s]
Packing dataset:  20%|█▉        | 9763/49172 [00:03<00:12, 3066.22it/s]
Packing dataset:  21%|██        | 10084/49172 [00:03<00:12, 3105.98it/s]
Packing dataset:  21%|██        | 10403/49172 [00:03<00:12, 3129.44it/s]
Packing dataset:  22%|██▏       | 10745/49172 [00:03<00:11, 3212.22it/s]
Packing dataset:  23%|██▎       | 11067/49172 [00:03<00:12, 3170.66it/s]
Packing dataset:  23%|██▎       | 11385/49172 [00:03<00:12, 3112.21it/s]
Packing dataset:  24%|██▍       | 11697/49172 [00:03<00:12, 3113.45it/s]
Packing dataset:  24%|██▍       | 12009/49172 [00:03<00:11, 3107.61it/s]
Packing dataset:  25%|██▌       | 12320/49172 [00:03<00:11, 3071.60it/s]
Packing dataset:  26%|██▌       | 12637/49172 [00:03<00:11, 3098.48it/s]
Packing dataset:  26%|██▋       | 12948/49172 [00:04<00:11, 3072.14it/s]
Packing dataset:  27%|██▋       | 13264/49172 [00:04<00:11, 3097.28it/s]
Packing dataset:  28%|██▊       | 13576/49172 [00:04<00:11, 3102.36it/s]
Packing dataset:  28%|██▊       | 13890/49172 [00:04<00:11, 3112.86it/s]
Packing dataset:  29%|██▉       | 14202/49172 [00:04<00:11, 3084.21it/s]
Packing dataset:  30%|██▉       | 14533/49172 [00:04<00:10, 3150.04it/s]
Packing dataset:  30%|███       | 14849/49172 [00:04<00:10, 3123.39it/s]
Packing dataset:  31%|███       | 15173/49172 [00:04<00:10, 3155.35it/s]
Packing dataset:  31%|███▏      | 15489/49172 [00:04<00:10, 3154.38it/s]
Packing dataset:  32%|███▏      | 15805/49172 [00:05<00:10, 3137.08it/s]
Packing dataset:  33%|███▎      | 16122/49172 [00:05<00:10, 3140.43it/s]
Packing dataset:  33%|███▎      | 16437/49172 [00:05<00:10, 3116.03it/s]
Packing dataset:  34%|███▍      | 16751/49172 [00:05<00:10, 3121.11it/s]
Packing dataset:  35%|███▍      | 17064/49172 [00:05<00:10, 3087.68it/s]
Packing dataset:  35%|███▌      | 17373/49172 [00:05<00:10, 3059.02it/s]
Packing dataset:  36%|███▌      | 17679/49172 [00:05<00:10, 3048.60it/s]
Packing dataset:  37%|███▋      | 17984/49172 [00:05<00:10, 3042.58it/s]
Packing dataset:  37%|███▋      | 18290/49172 [00:05<00:10, 3047.31it/s]
Packing dataset:  38%|███▊      | 18598/49172 [00:05<00:10, 3053.26it/s]
Packing dataset:  38%|███▊      | 18907/49172 [00:06<00:09, 3063.18it/s]
Packing dataset:  39%|███▉      | 19229/49172 [00:06<00:09, 3109.78it/s]
Packing dataset:  40%|███▉      | 19541/49172 [00:06<00:09, 3039.63it/s]
Packing dataset:  40%|████      | 19847/49172 [00:06<00:09, 3044.71it/s]
Packing dataset:  41%|████      | 20155/49172 [00:06<00:09, 3054.87it/s]
Packing dataset:  42%|████▏     | 20461/49172 [00:06<00:09, 3052.61it/s]
Packing dataset:  42%|████▏     | 20782/49172 [00:06<00:09, 3097.58it/s]
Packing dataset:  43%|████▎     | 21114/49172 [00:06<00:08, 3162.83it/s]
Packing dataset:  44%|████▎     | 21431/49172 [00:06<00:08, 3151.79it/s]
Packing dataset:  44%|████▍     | 21747/49172 [00:06<00:08, 3104.81it/s]
Packing dataset:  45%|████▍     | 22058/49172 [00:07<00:13, 2065.10it/s]
Packing dataset:  45%|████▌     | 22362/49172 [00:07<00:11, 2276.56it/s]
Packing dataset:  46%|████▌     | 22659/49172 [00:07<00:10, 2439.57it/s]
Packing dataset:  47%|████▋     | 22962/49172 [00:07<00:10, 2588.78it/s]
Packing dataset:  47%|████▋     | 23253/49172 [00:07<00:09, 2669.54it/s]
Packing dataset:  48%|████▊     | 23573/49172 [00:07<00:09, 2813.63it/s]
Packing dataset:  49%|████▊     | 23871/49172 [00:07<00:08, 2857.55it/s]
Packing dataset:  49%|████▉     | 24183/49172 [00:07<00:08, 2931.32it/s]
Packing dataset:  50%|████▉     | 24508/49172 [00:08<00:08, 3023.55it/s]
Packing dataset:  50%|█████     | 24816/49172 [00:08<00:08, 3030.26it/s]
Packing dataset:  51%|█████     | 25123/49172 [00:08<00:07, 3020.30it/s]
Packing dataset:  52%|█████▏    | 25441/49172 [00:08<00:07, 3062.78it/s]
Packing dataset:  52%|█████▏    | 25750/49172 [00:08<00:07, 3015.19it/s]
Packing dataset:  53%|█████▎    | 26055/49172 [00:08<00:07, 3022.30it/s]
Packing dataset:  54%|█████▎    | 26359/49172 [00:08<00:07, 3020.53it/s]
Packing dataset:  54%|█████▍    | 26671/49172 [00:08<00:07, 3047.71it/s]
Packing dataset:  55%|█████▍    | 26994/49172 [00:08<00:07, 3100.26it/s]
Packing dataset:  56%|█████▌    | 27315/49172 [00:08<00:06, 3129.91it/s]
Packing dataset:  56%|█████▌    | 27629/49172 [00:09<00:06, 3084.88it/s]
Packing dataset:  57%|█████▋    | 27938/49172 [00:09<00:06, 3057.58it/s]
Packing dataset:  57%|█████▋    | 28245/49172 [00:09<00:06, 3006.61it/s]
Packing dataset:  58%|█████▊    | 28546/49172 [00:09<00:06, 2985.00it/s]
Packing dataset:  59%|█████▊    | 28845/49172 [00:09<00:06, 2983.63it/s]
Packing dataset:  59%|█████▉    | 29150/49172 [00:09<00:06, 3000.84it/s]
Packing dataset:  60%|█████▉    | 29457/49172 [00:09<00:06, 3020.00it/s]
Packing dataset:  61%|██████    | 29764/49172 [00:09<00:06, 3033.38it/s]
Packing dataset:  61%|██████    | 30068/49172 [00:09<00:06, 3003.25it/s]
Packing dataset:  62%|██████▏   | 30386/49172 [00:09<00:06, 3052.80it/s]
Packing dataset:  62%|██████▏   | 30692/49172 [00:10<00:06, 3044.73it/s]
Packing dataset:  63%|██████▎   | 30997/49172 [00:10<00:05, 3035.08it/s]
Packing dataset:  64%|██████▎   | 31301/49172 [00:10<00:05, 3030.24it/s]
Packing dataset:  64%|██████▍   | 31627/49172 [00:10<00:05, 3098.33it/s]
Packing dataset:  65%|██████▍   | 31947/49172 [00:10<00:05, 3127.14it/s]
Packing dataset:  66%|██████▌   | 32260/49172 [00:10<00:05, 3086.10it/s]
Packing dataset:  66%|██████▌   | 32571/49172 [00:10<00:05, 3092.38it/s]
Packing dataset:  67%|██████▋   | 32898/49172 [00:10<00:05, 3143.91it/s]
Packing dataset:  68%|██████▊   | 33213/49172 [00:10<00:05, 3125.20it/s]
Packing dataset:  68%|██████▊   | 33537/49172 [00:10<00:04, 3159.10it/s]
Packing dataset:  69%|██████▉   | 33854/49172 [00:11<00:04, 3117.93it/s]
Packing dataset:  69%|██████▉   | 34166/49172 [00:11<00:04, 3111.70it/s]
Packing dataset:  70%|███████   | 34482/49172 [00:11<00:04, 3125.44it/s]
Packing dataset:  71%|███████   | 34795/49172 [00:11<00:04, 3119.97it/s]
Packing dataset:  71%|███████▏  | 35108/49172 [00:11<00:04, 3101.96it/s]
Packing dataset:  72%|███████▏  | 35445/49172 [00:11<00:04, 3179.09it/s]
Packing dataset:  73%|███████▎  | 35764/49172 [00:11<00:04, 3077.92it/s]
Packing dataset:  73%|███████▎  | 36073/49172 [00:11<00:04, 3053.83it/s]
Packing dataset:  74%|███████▍  | 36380/49172 [00:11<00:04, 3057.46it/s]
Packing dataset:  75%|███████▍  | 36707/49172 [00:11<00:04, 3115.27it/s]
Packing dataset:  75%|███████▌  | 37019/49172 [00:12<00:03, 3084.37it/s]
Packing dataset:  76%|███████▌  | 37337/49172 [00:12<00:03, 3111.06it/s]
Packing dataset:  77%|███████▋  | 37653/49172 [00:12<00:03, 3123.90it/s]
Packing dataset:  77%|███████▋  | 37971/49172 [00:12<00:03, 3140.55it/s]
Packing dataset:  78%|███████▊  | 38286/49172 [00:12<00:03, 3129.89it/s]
Packing dataset:  78%|███████▊  | 38600/49172 [00:12<00:03, 3067.57it/s]
Packing dataset:  79%|███████▉  | 38913/49172 [00:12<00:03, 3083.13it/s]
Packing dataset:  80%|███████▉  | 39231/49172 [00:12<00:03, 3108.91it/s]
Packing dataset:  80%|████████  | 39543/49172 [00:12<00:03, 3072.89it/s]
Packing dataset:  81%|████████  | 39851/49172 [00:12<00:03, 3072.29it/s]
Packing dataset:  82%|████████▏ | 40159/49172 [00:13<00:02, 3054.81it/s]
Packing dataset:  82%|████████▏ | 40472/49172 [00:13<00:02, 3074.50it/s]
Packing dataset:  83%|████████▎ | 40787/49172 [00:13<00:02, 3096.05it/s]
Packing dataset:  84%|████████▎ | 41102/49172 [00:13<00:02, 3110.04it/s]
Packing dataset:  84%|████████▍ | 41414/49172 [00:13<00:02, 3094.68it/s]
Packing dataset:  85%|████████▍ | 41724/49172 [00:13<00:02, 3080.54it/s]
Packing dataset:  85%|████████▌ | 42033/49172 [00:13<00:02, 3035.49it/s]
Packing dataset:  86%|████████▌ | 42337/49172 [00:13<00:02, 3027.51it/s]
Packing dataset:  87%|████████▋ | 42640/49172 [00:13<00:02, 2980.31it/s]
Packing dataset:  87%|████████▋ | 42943/49172 [00:14<00:02, 2994.07it/s]
Packing dataset:  88%|████████▊ | 43243/49172 [00:14<00:02, 2962.17it/s]
Packing dataset:  89%|████████▊ | 43544/49172 [00:14<00:01, 2975.34it/s]
Packing dataset:  89%|████████▉ | 43844/49172 [00:14<00:01, 2981.68it/s]
Packing dataset:  90%|████████▉ | 44145/49172 [00:14<00:01, 2985.92it/s]
Packing dataset:  90%|█████████ | 44460/49172 [00:14<00:01, 3032.18it/s]
Packing dataset:  91%|█████████ | 44772/49172 [00:14<00:01, 3057.26it/s]
Packing dataset:  92%|█████████▏| 45078/49172 [00:14<00:01, 3038.37it/s]
Packing dataset:  92%|█████████▏| 45408/49172 [00:14<00:01, 3113.45it/s]
Packing dataset:  93%|█████████▎| 45720/49172 [00:14<00:01, 3068.52it/s]
Packing dataset:  94%|█████████▎| 46028/49172 [00:15<00:01, 3021.73it/s]
Packing dataset:  94%|█████████▍| 46344/49172 [00:15<00:00, 3061.91it/s]
Packing dataset:  95%|█████████▍| 46651/49172 [00:15<00:00, 3055.09it/s]
Packing dataset:  96%|█████████▌| 46964/49172 [00:15<00:00, 3076.03it/s]
Packing dataset:  96%|█████████▌| 47272/49172 [00:15<00:00, 3036.18it/s]
Packing dataset:  97%|█████████▋| 47586/49172 [00:15<00:00, 3066.01it/s]
Packing dataset:  97%|█████████▋| 47893/49172 [00:15<00:00, 3045.69it/s]
Packing dataset:  98%|█████████▊| 48198/49172 [00:15<00:00, 3039.14it/s]
Packing dataset:  99%|█████████▊| 48504/49172 [00:15<00:00, 3042.55it/s]
Packing dataset:  99%|█████████▉| 48822/49172 [00:15<00:00, 3082.53it/s]
Packing dataset: 100%|█████████▉| 49131/49172 [00:16<00:00, 3079.27it/s]
Packing dataset: 100%|██████████| 49172/49172 [00:16<00:00, 3063.88it/s]
Learning rate scheduler is initialized.
 Profiling disabled.
 Profiler config after instantiation: {'enabled': False}

  0%|          | 0/10 [00:00<?, ?it/s]/home/jenkins/xiangdong/torchtune/recipes/lora_finetune_distributed.py:724: FutureWarning: scale_grads is deprecated and will be removed in future versions. Please use `scale_grads_` instead.
  training.scale_grads(self._model, self.world_size / num_tokens)

 10%|█         | 1/10 [00:25<03:52, 25.80s/it]
1|1|Loss: 1.7554633617401123:  10%|█         | 1/10 [00:25<03:52, 25.80s/it]iteration:  1 tokens:  12514 time:  25.79809158714488 tokens_per_second:  485.07

1|1|Loss: 1.7554633617401123:  20%|██        | 2/10 [00:36<02:14, 16.80s/it]
1|2|Loss: 1.7078171968460083:  20%|██        | 2/10 [00:36<02:14, 16.80s/it]iteration:  2 tokens:  12136 time:  10.469081602059305 tokens_per_second:  1159.22

1|2|Loss: 1.7078171968460083:  30%|███       | 3/10 [00:46<01:36, 13.84s/it]
1|3|Loss: 1.713700532913208:  30%|███       | 3/10 [00:46<01:36, 13.84s/it] iteration:  3 tokens:  12520 time:  10.290417295880616 tokens_per_second:  1216.67

1|3|Loss: 1.713700532913208:  40%|████      | 4/10 [00:56<01:14, 12.41s/it]
1|4|Loss: 1.781863808631897:  40%|████      | 4/10 [00:56<01:14, 12.41s/it]iteration:  4 tokens:  11506 time:  10.189309824025258 tokens_per_second:  1129.22

1|4|Loss: 1.781863808631897:  50%|█████     | 5/10 [01:07<00:58, 11.65s/it]
1|5|Loss: 1.714793086051941:  50%|█████     | 5/10 [01:07<00:58, 11.65s/it]iteration:  5 tokens:  12732 time:  10.278056262992322 tokens_per_second:  1238.76

1|5|Loss: 1.714793086051941:  60%|██████    | 6/10 [01:17<00:44, 11.20s/it]
1|6|Loss: 1.7106053829193115:  60%|██████    | 6/10 [01:17<00:44, 11.20s/it]iteration:  6 tokens:  13067 time:  10.297820094041526 tokens_per_second:  1268.91

1|6|Loss: 1.7106053829193115:  70%|███████   | 7/10 [01:27<00:32, 10.90s/it]
1|7|Loss: 1.829026460647583:  70%|███████   | 7/10 [01:27<00:32, 10.90s/it] iteration:  7 tokens:  12311 time:  10.261304416926578 tokens_per_second:  1199.75

1|7|Loss: 1.829026460647583:  80%|████████  | 8/10 [01:38<00:21, 10.71s/it]
1|8|Loss: 1.7280372381210327:  80%|████████  | 8/10 [01:38<00:21, 10.71s/it]iteration:  8 tokens:  12292 time:  10.284754071151838 tokens_per_second:  1195.17

1|8|Loss: 1.7280372381210327:  90%|█████████ | 9/10 [01:48<00:10, 10.59s/it]
1|9|Loss: 1.7175650596618652:  90%|█████████ | 9/10 [01:48<00:10, 10.59s/it]iteration:  9 tokens:  12768 time:  10.29263400612399 tokens_per_second:  1240.5

1|9|Loss: 1.7175650596618652: 100%|██████████| 10/10 [01:58<00:00, 10.51s/it]
1|10|Loss: 1.6275007724761963: 100%|██████████| 10/10 [01:58<00:00, 10.51s/it]iteration:  10 tokens:  13657 time:  10.32463634898886 tokens_per_second:  1322.76
[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/jenkins/xiangdong/torchtune/recipes/lora_finetune_distributed.py", line 912, in <module>
[rank1]:     sys.exit(recipe_main())
[rank1]:   File "/home/jenkins/xiangdong/torchtune/torchtune/config/_parse.py", line 99, in wrapper
[rank1]:     sys.exit(recipe_main(conf))
[rank1]:   File "/home/jenkins/xiangdong/torchtune/recipes/lora_finetune_distributed.py", line 907, in recipe_main
[rank1]:     recipe.train()
[rank1]:   File "/home/jenkins/xiangdong/torchtune/recipes/lora_finetune_distributed.py", line 808, in train
[rank1]:     print("avg tokens_per_second: ", round(total_tokens / total_time, 2))
[rank1]: ZeroDivisionError: division by zero
avg tokens_per_second:  1226.64

1|10|Loss: 1.6275007724761963: 100%|██████████| 10/10 [01:58<00:00, 11.87s/it]
[rank0]:[W1020 09:27:33.889499587 ProcessGroup.hpp:940] Warning: No backend of type 0 found for Process Group with name undefined. Assuming no hooks are registered. (function hasHooks)
W1020 09:27:34.094000 2402310 site-packages/torch/distributed/elastic/multiprocessing/api.py:940] Sending process 2402384 closing signal SIGTERM
E1020 09:27:34.208000 2402310 site-packages/torch/distributed/elastic/multiprocessing/api.py:914] failed (exitcode: 1) local_rank: 1 (pid: 2402385) of binary: /home/jenkins/.conda/envs/xpu_op_/bin/python3.10
Running with torchrun...
Traceback (most recent call last):
  File "/home/jenkins/.conda/envs/xpu_op_/bin/tune", line 7, in <module>
    sys.exit(main())
  File "/home/jenkins/xiangdong/torchtune/torchtune/_cli/tune.py", line 52, in main
    parser.run(args)
  File "/home/jenkins/xiangdong/torchtune/torchtune/_cli/tune.py", line 46, in run
    args.func(args)
  File "/home/jenkins/xiangdong/torchtune/torchtune/_cli/run.py", line 212, in _run_cmd
    self._run_distributed(args, is_builtin=is_builtin)
  File "/home/jenkins/.conda/envs/xpu_op_/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 358, in wrapper
    return f(*args, **kwargs)
  File "/home/jenkins/xiangdong/torchtune/torchtune/_cli/run.py", line 101, in _run_distributed
    run(args)
  File "/home/jenkins/.conda/envs/xpu_op_/lib/python3.10/site-packages/torch/distributed/run.py", line 937, in run
    elastic_launch(
  File "/home/jenkins/.conda/envs/xpu_op_/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 159, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/jenkins/.conda/envs/xpu_op_/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 300, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/home/jenkins/xiangdong/torchtune/recipes/lora_finetune_distributed.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2025-10-20_09:27:34
  host      : dut7358
  rank      : 0 (local_rank: 0)
  exitcode  : -15 (pid: 2402384)
  error_file: <N/A>
  traceback : Signal 15 (SIGTERM) received by PID 2402384
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-10-20_09:27:34
  host      : dut7358
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 2402385)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================