-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathMeta-Llama-3.1-8B-Instruct_lora_single.log
More file actions
302 lines (296 loc) · 19.8 KB
/
Copy pathMeta-Llama-3.1-8B-Instruct_lora_single.log
File metadata and controls
302 lines (296 loc) · 19.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
Running LoRAFinetuneRecipeSingleDevice with resolved config:
batch_size: 2
batch_size_val: 2
checkpointer:
_component_: torchtune.training.FullModelHFCheckpointer
checkpoint_dir: /tmp/Meta-Llama-3.1-8B-Instruct/
checkpoint_files:
- model-00001-of-00004.safetensors
- model-00002-of-00004.safetensors
- model-00003-of-00004.safetensors
- model-00004-of-00004.safetensors
model_type: LLAMA3
output_dir: /tmp/torchtune/llama3_1_8B/lora
recipe_checkpoint: null
clip_grad_norm: null
compile: false
dataset:
_component_: torchtune.datasets.alpaca_cleaned_dataset
packed: true
split: train[:95%]
dataset_val:
_component_: torchtune.datasets.alpaca_cleaned_dataset
split: train[95%:]
device: xpu
dtype: bf16
enable_activation_checkpointing: false
enable_activation_offloading: false
epochs: 1
gradient_accumulation_steps: 8
log_every_n_steps: 1
log_level: INFO
log_peak_memory_stats: true
loss:
_component_: torchtune.modules.loss.LinearCrossEntropyLoss
lr_scheduler:
_component_: torchtune.training.lr_schedulers.get_cosine_schedule_with_warmup
num_warmup_steps: 100
max_steps_per_epoch: 10
metric_logger:
_component_: torchtune.training.metric_logging.DiskLogger
log_dir: /tmp/torchtune/llama3_1_8B/lora/logs
model:
_component_: torchtune.models.llama3_1.lora_llama3_1_8b
apply_lora_to_mlp: true
apply_lora_to_output: false
lora_alpha: 16
lora_attn_modules:
- q_proj
- v_proj
- output_proj
lora_dropout: 0.0
lora_rank: 8
optimizer:
_component_: torch.optim.AdamW
fused: true
lr: 0.0003
weight_decay: 0.01
output_dir: /tmp/torchtune/llama3_1_8B/lora
profiler:
_component_: torchtune.training.setup_torch_profiler
active_steps: 2
cpu: true
cuda: true
enabled: false
num_cycles: 1
output_dir: /tmp/torchtune/llama3_1_8B/lora/profiling_outputs
profile_memory: false
record_shapes: true
wait_steps: 5
warmup_steps: 3
with_flops: false
with_stack: false
resume_from_checkpoint: false
run_val_every_n_steps: null
save_adapter_weights_only: false
seed: 123
shuffle: true
tokenizer:
_component_: torchtune.models.llama3.llama3_tokenizer
max_seq_len: 512
path: /tmp/Meta-Llama-3.1-8B-Instruct/original/tokenizer.model
/home/jenkins/xiangdong/torchtune/recipes/lora_finetune_single_device.py:436: FutureWarning: lora_attn_modules is deprecated for validate_missing_and_unexpected_for_lora and will be removed in future versions. Please use state_dict_keys instead.
validate_missing_and_unexpected_for_lora(
/home/jenkins/xiangdong/torchtune/torchtune/utils/_logging.py:143: FutureWarning: apply_lora_to_mlp is deprecated for validate_missing_and_unexpected_for_lora and will be removed in future versions. Please use state_dict_keys instead.
return obj(*args, **kwargs)
/home/jenkins/xiangdong/torchtune/torchtune/utils/_logging.py:143: FutureWarning: apply_lora_to_output is deprecated for validate_missing_and_unexpected_for_lora and will be removed in future versions. Please use state_dict_keys instead.
return obj(*args, **kwargs)
Model is initialized with precision torch.bfloat16.
Memory stats after model init:
XPU peak memory active: 15.06 GiB
XPU peak memory alloc: 15.06 GiB
XPU peak memory reserved: 15.18 GiB
Tokenizer is initialized from file.
Optimizer and loss are initialized.
Loss is initialized.
Writing logs to /tmp/torchtune/llama3_1_8B/lora/logs/log_1760954924.txt
Packing dataset: 0%| | 0/49172 [00:00<?, ?it/s]Packing dataset: 0%| | 212/49172 [00:00<00:23, 2119.26it/s]Packing dataset: 1%| | 462/49172 [00:00<00:20, 2335.37it/s]Packing dataset: 2%|▏ | 739/49172 [00:00<00:19, 2532.94it/s]Packing dataset: 2%|▏ | 1047/49172 [00:00<00:17, 2746.61it/s]Packing dataset: 3%|▎ | 1367/49172 [00:00<00:16, 2909.54it/s]Packing dataset: 3%|▎ | 1664/49172 [00:00<00:16, 2929.63it/s]Packing dataset: 4%|▍ | 1960/49172 [00:00<00:16, 2939.49it/s]Packing dataset: 5%|▍ | 2284/49172 [00:00<00:15, 3032.79it/s]Packing dataset: 5%|▌ | 2613/49172 [00:00<00:14, 3110.59it/s]Packing dataset: 6%|▌ | 2942/49172 [00:01<00:14, 3163.17it/s]Packing dataset: 7%|▋ | 3259/49172 [00:01<00:14, 3160.24it/s]Packing dataset: 7%|▋ | 3576/49172 [00:01<00:14, 3136.87it/s]Packing dataset: 8%|▊ | 3897/49172 [00:01<00:14, 3157.29it/s]Packing dataset: 9%|▊ | 4213/49172 [00:01<00:14, 3121.32it/s]Packing dataset: 9%|▉ | 4527/49172 [00:01<00:14, 3124.16it/s]Packing dataset: 10%|▉ | 4840/49172 [00:01<00:14, 3112.43it/s]Packing dataset: 10%|█ | 5154/49172 [00:01<00:14, 3119.98it/s]Packing dataset: 11%|█ | 5467/49172 [00:01<00:14, 3113.39it/s]Packing dataset: 12%|█▏ | 5800/49172 [00:01<00:13, 3175.51it/s]Packing dataset: 12%|█▏ | 6118/49172 [00:02<00:13, 3129.37it/s]Packing dataset: 13%|█▎ | 6432/49172 [00:02<00:13, 3123.30it/s]Packing dataset: 14%|█▎ | 6745/49172 [00:02<00:13, 3054.07it/s]Packing dataset: 14%|█▍ | 7051/49172 [00:02<00:13, 3047.06it/s]Packing dataset: 15%|█▍ | 7356/49172 [00:02<00:13, 3045.76it/s]Packing dataset: 16%|█▌ | 7679/49172 [00:02<00:13, 3098.81it/s]Packing dataset: 16%|█▌ | 7990/49172 [00:02<00:13, 3037.31it/s]Packing dataset: 17%|█▋ | 8295/49172 [00:02<00:13, 3013.69it/s]Packing dataset: 18%|█▊ | 8611/49172 [00:02<00:13, 3056.16it/s]Packing dataset: 18%|█▊ | 8934/49172 [00:02<00:12, 3106.57it/s]Packing dataset: 19%|█▉ | 9245/49172 [00:03<00:12, 3091.40it/s]Packing dataset: 19%|█▉ | 9555/49172 [00:03<00:13, 3033.32it/s]Packing dataset: 20%|██ | 9859/49172 [00:03<00:13, 3001.37it/s]Packing dataset: 21%|██ | 10177/49172 [00:03<00:12, 3053.58it/s]Packing dataset: 21%|██▏ | 10486/49172 [00:03<00:12, 3062.95it/s]Packing dataset: 22%|██▏ | 10809/49172 [00:03<00:12, 3108.26it/s]Packing dataset: 23%|██▎ | 11121/49172 [00:03<00:12, 3067.92it/s]Packing dataset: 23%|██▎ | 11429/49172 [00:03<00:12, 3045.54it/s]Packing dataset: 24%|██▍ | 11734/49172 [00:03<00:12, 3042.61it/s]Packing dataset: 24%|██▍ | 12039/49172 [00:03<00:12, 3024.45it/s]Packing dataset: 25%|██▌ | 12342/49172 [00:04<00:12, 3022.46it/s]Packing dataset: 26%|██▌ | 12649/49172 [00:04<00:12, 3036.37it/s]Packing dataset: 26%|██▋ | 12953/49172 [00:04<00:12, 3007.82it/s]Packing dataset: 27%|██▋ | 13258/49172 [00:04<00:11, 3019.69it/s]Packing dataset: 28%|██▊ | 13561/49172 [00:04<00:11, 3015.79it/s]Packing dataset: 28%|██▊ | 13865/49172 [00:04<00:11, 3021.70it/s]Packing dataset: 29%|██▉ | 14168/49172 [00:04<00:11, 2973.03it/s]Packing dataset: 29%|██▉ | 14483/49172 [00:04<00:11, 3024.70it/s]Packing dataset: 30%|███ | 14786/49172 [00:04<00:11, 2988.64it/s]Packing dataset: 31%|███ | 15091/49172 [00:04<00:11, 3004.60it/s]Packing dataset: 31%|███▏ | 15410/49172 [00:05<00:11, 3057.06it/s]Packing dataset: 32%|███▏ | 15730/49172 [00:05<00:10, 3099.14it/s]Packing dataset: 33%|███▎ | 16049/49172 [00:05<00:10, 3126.21it/s]Packing dataset: 33%|███▎ | 16362/49172 [00:05<00:10, 3035.59it/s]Packing dataset: 34%|███▍ | 16674/49172 [00:05<00:10, 3057.82it/s]Packing dataset: 35%|███▍ | 16981/49172 [00:05<00:10, 3026.46it/s]Packing dataset: 35%|███▌ | 17286/49172 [00:05<00:10, 3030.05it/s]Packing dataset: 36%|███▌ | 17590/49172 [00:05<00:10, 2993.01it/s]Packing dataset: 36%|███▋ | 17890/49172 [00:05<00:10, 2963.76it/s]Packing dataset: 37%|███▋ | 18194/49172 [00:05<00:10, 2983.68it/s]Packing dataset: 38%|███▊ | 18493/49172 [00:06<00:10, 2955.49it/s]Packing dataset: 38%|███▊ | 18812/49172 [00:06<00:10, 3024.38it/s]Packing dataset: 39%|███▉ | 19115/49172 [00:06<00:09, 3010.09it/s]Packing dataset: 39%|███▉ | 19417/49172 [00:06<00:09, 2976.05it/s]Packing dataset: 40%|████ | 19715/49172 [00:06<00:10, 2942.62it/s]Packing dataset: 41%|████ | 20010/49172 [00:06<00:09, 2925.80it/s]Packing dataset: 41%|████▏ | 20311/49172 [00:06<00:09, 2949.10it/s]Packing dataset: 42%|████▏ | 20607/49172 [00:06<00:09, 2926.92it/s]Packing dataset: 43%|████▎ | 20924/49172 [00:06<00:09, 2989.06it/s]Packing dataset: 43%|████▎ | 21245/49172 [00:07<00:09, 3049.72it/s]Packing dataset: 44%|████▍ | 21551/49172 [00:07<00:09, 3039.96it/s]Packing dataset: 44%|████▍ | 21861/49172 [00:07<00:08, 3054.36it/s]Packing dataset: 45%|████▌ | 22167/49172 [00:07<00:09, 2969.86it/s]Packing dataset: 46%|████▌ | 22487/49172 [00:07<00:08, 3031.34it/s]Packing dataset: 46%|████▋ | 22791/49172 [00:07<00:08, 2976.28it/s]Packing dataset: 47%|████▋ | 23090/49172 [00:07<00:08, 2972.32it/s]Packing dataset: 48%|████▊ | 23403/49172 [00:07<00:08, 3018.30it/s]Packing dataset: 48%|████▊ | 23718/49172 [00:07<00:08, 3056.43it/s]Packing dataset: 49%|████▉ | 24028/49172 [00:07<00:08, 3066.98it/s]Packing dataset: 50%|████▉ | 24342/49172 [00:08<00:08, 3087.19it/s]Packing dataset: 50%|█████ | 24665/49172 [00:08<00:07, 3128.82it/s]Packing dataset: 51%|█████ | 24979/49172 [00:08<00:07, 3064.56it/s]Packing dataset: 51%|█████▏ | 25286/49172 [00:08<00:07, 3064.86it/s]Packing dataset: 52%|█████▏ | 25594/49172 [00:08<00:07, 3064.41it/s]Packing dataset: 53%|█████▎ | 25901/49172 [00:08<00:07, 3011.07it/s]Packing dataset: 53%|█████▎ | 26226/49172 [00:08<00:07, 3078.32it/s]Packing dataset: 54%|█████▍ | 26535/49172 [00:08<00:07, 3040.50it/s]Packing dataset: 55%|█████▍ | 26840/49172 [00:08<00:07, 3000.71it/s]Packing dataset: 55%|█████▌ | 27150/49172 [00:08<00:07, 3029.37it/s]Packing dataset: 56%|█████▌ | 27458/49172 [00:09<00:07, 3042.60it/s]Packing dataset: 56%|█████▋ | 27763/49172 [00:09<00:07, 3000.81it/s]Packing dataset: 57%|█████▋ | 28064/49172 [00:09<00:07, 2955.67it/s]Packing dataset: 58%|█████▊ | 28365/49172 [00:09<00:07, 2969.15it/s]Packing dataset: 58%|█████▊ | 28663/49172 [00:09<00:06, 2931.73it/s]Packing dataset: 59%|█████▉ | 28959/49172 [00:09<00:06, 2937.00it/s]Packing dataset: 59%|█████▉ | 29256/49172 [00:09<00:06, 2945.59it/s]Packing dataset: 60%|██████ | 29560/49172 [00:09<00:06, 2968.35it/s]Packing dataset: 61%|██████ | 29858/49172 [00:09<00:06, 2969.15it/s]Packing dataset: 61%|██████▏ | 30165/49172 [00:09<00:06, 2998.47it/s]Packing dataset: 62%|██████▏ | 30465/49172 [00:10<00:06, 2966.34it/s]Packing dataset: 63%|██████▎ | 30767/49172 [00:10<00:06, 2978.36it/s]Packing dataset: 63%|██████▎ | 31065/49172 [00:10<00:06, 2968.96it/s]Packing dataset: 64%|██████▍ | 31363/49172 [00:10<00:05, 2971.81it/s]Packing dataset: 64%|██████▍ | 31698/49172 [00:10<00:05, 3082.65it/s]Packing dataset: 65%|██████▌ | 32007/49172 [00:10<00:05, 3049.31it/s]Packing dataset: 66%|██████▌ | 32313/49172 [00:10<00:05, 3035.95it/s]Packing dataset: 66%|██████▋ | 32619/49172 [00:10<00:05, 3039.44it/s]Packing dataset: 67%|██████▋ | 32927/49172 [00:10<00:05, 3051.15it/s]Packing dataset: 68%|██████▊ | 33239/49172 [00:10<00:05, 3069.22it/s]Packing dataset: 68%|██████▊ | 33547/49172 [00:11<00:05, 3072.09it/s]Packing dataset: 69%|██████▉ | 33855/49172 [00:11<00:07, 2076.81it/s]Packing dataset: 69%|██████▉ | 34155/49172 [00:11<00:06, 2282.96it/s]Packing dataset: 70%|███████ | 34447/49172 [00:11<00:06, 2436.10it/s]Packing dataset: 71%|███████ | 34752/49172 [00:11<00:05, 2591.02it/s]Packing dataset: 71%|███████▏ | 35055/49172 [00:11<00:05, 2704.01it/s]Packing dataset: 72%|███████▏ | 35366/49172 [00:11<00:04, 2811.88it/s]Packing dataset: 73%|███████▎ | 35661/49172 [00:11<00:04, 2834.57it/s]Packing dataset: 73%|███████▎ | 35954/49172 [00:12<00:04, 2836.22it/s]Packing dataset: 74%|███████▎ | 36245/49172 [00:12<00:04, 2840.10it/s]Packing dataset: 74%|███████▍ | 36566/49172 [00:12<00:04, 2943.76it/s]Packing dataset: 75%|███████▍ | 36864/49172 [00:12<00:04, 2945.60it/s]Packing dataset: 76%|███████▌ | 37167/49172 [00:12<00:04, 2966.17it/s]Packing dataset: 76%|███████▌ | 37467/49172 [00:12<00:03, 2973.91it/s]Packing dataset: 77%|███████▋ | 37770/49172 [00:12<00:03, 2986.37it/s]Packing dataset: 77%|███████▋ | 38075/49172 [00:12<00:03, 3000.33it/s]Packing dataset: 78%|███████▊ | 38376/49172 [00:12<00:03, 2983.42it/s]Packing dataset: 79%|███████▊ | 38675/49172 [00:12<00:03, 2949.52it/s]Packing dataset: 79%|███████▉ | 38971/49172 [00:13<00:03, 2907.04it/s]Packing dataset: 80%|███████▉ | 39282/49172 [00:13<00:03, 2966.43it/s]Packing dataset: 80%|████████ | 39580/49172 [00:13<00:03, 2956.15it/s]Packing dataset: 81%|████████ | 39879/49172 [00:13<00:03, 2964.10it/s]Packing dataset: 82%|████████▏ | 40176/49172 [00:13<00:03, 2936.28it/s]Packing dataset: 82%|████████▏ | 40487/49172 [00:13<00:02, 2985.17it/s]Packing dataset: 83%|████████▎ | 40793/49172 [00:13<00:02, 3005.61it/s]Packing dataset: 84%|████████▎ | 41094/49172 [00:13<00:02, 3003.95it/s]Packing dataset: 84%|████████▍ | 41400/49172 [00:13<00:02, 3020.31it/s]Packing dataset: 85%|████████▍ | 41703/49172 [00:13<00:02, 2976.88it/s]Packing dataset: 85%|████████▌ | 42001/49172 [00:14<00:02, 2927.62it/s]Packing dataset: 86%|████████▌ | 42302/49172 [00:14<00:02, 2946.89it/s]Packing dataset: 87%|████████▋ | 42597/49172 [00:14<00:02, 2926.91it/s]Packing dataset: 87%|████████▋ | 42896/49172 [00:14<00:02, 2944.02it/s]Packing dataset: 88%|████████▊ | 43191/49172 [00:14<00:02, 2904.41it/s]Packing dataset: 88%|████████▊ | 43486/49172 [00:14<00:01, 2915.14it/s]Packing dataset: 89%|████████▉ | 43785/49172 [00:14<00:01, 2936.18it/s]Packing dataset: 90%|████████▉ | 44079/49172 [00:14<00:01, 2910.99it/s]Packing dataset: 90%|█████████ | 44384/49172 [00:14<00:01, 2949.62it/s]Packing dataset: 91%|█████████ | 44680/49172 [00:15<00:01, 2915.65it/s]Packing dataset: 91%|█████████▏| 44984/49172 [00:15<00:01, 2949.77it/s]Packing dataset: 92%|█████████▏| 45289/49172 [00:15<00:01, 2977.79it/s]Packing dataset: 93%|█████████▎| 45604/49172 [00:15<00:01, 3025.57it/s]Packing dataset: 93%|█████████▎| 45907/49172 [00:15<00:01, 2926.48it/s]Packing dataset: 94%|█████████▍| 46204/49172 [00:15<00:01, 2938.07it/s]Packing dataset: 95%|█████████▍| 46500/49172 [00:15<00:00, 2940.71it/s]Packing dataset: 95%|█████████▌| 46812/49172 [00:15<00:00, 2993.32it/s]Packing dataset: 96%|█████████▌| 47113/49172 [00:15<00:00, 2996.14it/s]Packing dataset: 96%|█████████▋| 47413/49172 [00:15<00:00, 2971.20it/s]Packing dataset: 97%|█████████▋| 47711/49172 [00:16<00:00, 2944.40it/s]Packing dataset: 98%|█████████▊| 48013/49172 [00:16<00:00, 2964.32it/s]Packing dataset: 98%|█████████▊| 48310/49172 [00:16<00:00, 2947.84it/s]Packing dataset: 99%|█████████▉| 48616/49172 [00:16<00:00, 2980.15it/s]Packing dataset: 99%|█████████▉| 48917/49172 [00:16<00:00, 2989.00it/s]Packing dataset: 100%|██████████| 49172/49172 [00:16<00:00, 2977.19it/s]
Learning rate scheduler is initialized.
Profiling disabled.
Profiler config after instantiation: {'enabled': False}
0%| | 0/10 [00:00<?, ?it/s]/home/jenkins/xiangdong/torchtune/recipes/lora_finetune_single_device.py:626: FutureWarning: scale_grads is deprecated and will be removed in future versions. Please use `scale_grads_` instead.
training.scale_grads(self._model, 1 / num_tokens)
10%|█ | 1/10 [00:05<00:46, 5.22s/it]1|1|Loss: 1.7901911735534668: 10%|█ | 1/10 [00:05<00:46, 5.22s/it]1|1|Loss: 1.7901911735534668: 20%|██ | 2/10 [00:09<00:35, 4.48s/it]1|2|Loss: 1.7191609144210815: 20%|██ | 2/10 [00:09<00:35, 4.48s/it]1|2|Loss: 1.7191609144210815: 30%|███ | 3/10 [00:13<00:29, 4.20s/it]1|3|Loss: 1.5265789031982422: 30%|███ | 3/10 [00:13<00:29, 4.20s/it]1|3|Loss: 1.5265789031982422: 40%|████ | 4/10 [00:16<00:24, 4.06s/it]1|4|Loss: 1.897816777229309: 40%|████ | 4/10 [00:16<00:24, 4.06s/it] 1|4|Loss: 1.897816777229309: 50%|█████ | 5/10 [00:20<00:19, 3.98s/it]1|5|Loss: 1.633756160736084: 50%|█████ | 5/10 [00:20<00:19, 3.98s/it]1|5|Loss: 1.633756160736084: 60%|██████ | 6/10 [00:24<00:15, 3.95s/it]1|6|Loss: 1.7722874879837036: 60%|██████ | 6/10 [00:24<00:15, 3.95s/it]1|6|Loss: 1.7722874879837036: 70%|███████ | 7/10 [00:28<00:11, 3.92s/it]1|7|Loss: 1.791071891784668: 70%|███████ | 7/10 [00:28<00:11, 3.92s/it] 1|7|Loss: 1.791071891784668: 80%|████████ | 8/10 [00:32<00:07, 3.87s/it]1|8|Loss: 1.7200340032577515: 80%|████████ | 8/10 [00:32<00:07, 3.87s/it]1|8|Loss: 1.7200340032577515: 90%|█████████ | 9/10 [00:36<00:03, 3.87s/it]1|9|Loss: 1.6448132991790771: 90%|█████████ | 9/10 [00:36<00:03, 3.87s/it]1|9|Loss: 1.6448132991790771: 100%|██████████| 10/10 [00:39<00:00, 3.87s/it]1|10|Loss: 1.6628748178482056: 100%|██████████| 10/10 [00:39<00:00, 3.87s/it]Starting checkpoint save...
Checkpoint saved in 0.00 seconds.
1|10|Loss: 1.6628748178482056: 100%|██████████| 10/10 [00:40<00:00, 4.00s/it]
iteration: 1 tokens: 6269 time: 5.221715496154502 tokens_per_second_on_single_device: 1200.56
iteration: 2 tokens: 6245 time: 3.9217470318544656 tokens_per_second_on_single_device: 1592.4
iteration: 3 tokens: 6258 time: 3.8509285419713706 tokens_per_second_on_single_device: 1625.06
iteration: 4 tokens: 5878 time: 3.824023633962497 tokens_per_second_on_single_device: 1537.12
iteration: 5 tokens: 6035 time: 3.823235129006207 tokens_per_second_on_single_device: 1578.51
iteration: 6 tokens: 6485 time: 3.86237626709044 tokens_per_second_on_single_device: 1679.02
iteration: 7 tokens: 6039 time: 3.82378999190405 tokens_per_second_on_single_device: 1579.32
iteration: 8 tokens: 5467 time: 3.7668837741948664 tokens_per_second_on_single_device: 1451.33
iteration: 9 tokens: 6381 time: 3.8503869029227644 tokens_per_second_on_single_device: 1657.24
iteration: 10 tokens: 6351 time: 3.8551411191001534 tokens_per_second_on_single_device: 1647.41
avg tokens_per_second_on_single_device: 1594.88