If new tokens are added to the tokenizer, should I invoke the resize_token_embeddings method?

[ChrisKimZHT]

Refer to #29 , tokenizer size and embedding size mismatch due to padding.

So if I only add 4 new tokens to the tokenizer and tokenizer size still smaller than embedding size, should I invoke the resize_token_embeddings method?

print(len(tokenizer)) # 151665
new_tokens = ["<think>", "</think>", "<answer>", "</answer>"]
tokenizer.add_tokens(new_tokens)
model.resize_token_embeddings(len(tokenizer)) # vocab_size: 152064, should I call this?
print(len(tokenizer)) # 151669

[ChrisKimZHT]

I'm performing SFT on Qwen2.5-Coder-7B using trl, below is my training config.

dataset_name: "../data/dataset_sft.jsonl"

model_name_or_path: "/home/chriskim/Qwen2.5-Coder-7B"
torch_dtype: bfloat16
trust_remote_code: true
attn_implementation: flash_attention_2
load_in_8bit: false
load_in_4bit: false
bnb_4bit_quant_type: nf4
use_bnb_nested_quant: true

use_peft: true
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - embed_tokens
  - lm_head
  - q_proj
  - v_proj
lora_task_type: "CAUSAL_LM"

output_dir: "./results"
per_device_train_batch_size: 1
gradient_accumulation_steps: 1
learning_rate: 5e-6
num_train_epochs: 10
lr_scheduler_type: cosine
warmup_ratio: 0.1
log_on_each_node: false
logging_first_step: true
logging_steps: 50
save_steps: 1000
save_total_limit: 10
seed: 42
bf16: true
optim: adamw_torch
report_to: tensorboard
gradient_checkpointing: true

dataset_num_proc: 16
max_seq_length: 10000
packing: true

[jklj077]

I'm not familiar with trl. but in general, it is not necessary. certain finetuning frameworks also resize the token embeddings automatically based on len(tokenizer). if it is indeed needed, there will definitely be errors.