Question About Hidden Space Tokenization Decoding

[Lingy12]

multilingual_analysis/layers/test_layer.py

Line 44 in be2b161

hidden_states, outputs = model.generate(**{'input_ids':inputs.input_ids, 'max_new_tokens':64})

I am trying to reproduce the token analysis of layers' hidden embedding. I have done the following:

1. Install according to requirements.txt
2. Replace the generation and models folder of transformers with the one in the repository

However, the output of the line is just the decoded token ids, no hidden_states (as normal generate).

May I ask is there any thing I am missing? Thank you very much.

[daoyangl]

Hello, did you solve this problem? I also have the same question about test_layer.py code.

[james-yw]

Hello, I have encountered the same issue. Did you manage to resolve it?

[Mongjin]

Hello, I encountered the same issue, but I was able to solve it by revising the Prompting function as shown below.
First of all, sorry for the hard-coded solution...
@Lingy12 @daoyangl @james-yw

If you think that my code may cause an error (i.e., be incorrect), please let me know!

def Prompting(model, prompt, candidate_premature_layers):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(inputs.input_ids, max_new_tokens=64, return_dict_in_generate=True, output_hidden_states=True)
    # hidden_states, outputs = model.generate(**{'input_ids':inputs.input_ids})
    hidden_states = outputs.hidden_states[1:][1:]
    hidden_embed = {}
    hidden_embed_token_level = {}
    lm_head = model.get_output_embeddings()
    for i, early_exit_layer in enumerate(candidate_premature_layers):
        layer_hidden_state = hidden_states[:][early_exit_layer]
        layer_hidden_state = torch.cat(layer_hidden_state, dim=1)
        layer_hidden_logits = lm_head(layer_hidden_state)
        layer_hidden_unembedding = torch.nn.functional.softmax(layer_hidden_logits, dim=-1)
        values, indicies = torch.topk(layer_hidden_unembedding, 100)
        indicies = indicies[0][0][:]
        hidden_embed[early_exit_layer] = tokenizer.decode(indicies)
        hidden_embed_token_level[early_exit_layer] = [tokenizer.decode(tok) for tok in indicies]
    answer = tokenizer.decode(outputs.sequences.tolist()[0]).replace('<pad> ', '')
    answer = answer.replace('</s>', '')
    
    return hidden_embed, hidden_embed_token_level, answer