Unable to import local transformers folder

[RadiantCrystal]

Hi @zhaoyiran924 ,

Very nice work! I am also getting the same issue. Can you please help me to import your local transformers package instead of directly installed 'transformers' package? When I am trying to import it from local folder it is showing the below error

Traceback (most recent call last): File "/media/rimi/DATA/Rima/multilingual_analysis/neuron_detection/neuron_detection.py", line 17, in from transformers import AutoModelForCausalLM, AutoTokenizer ImportError: cannot import name 'AutoModelForCausalLM' from 'transformers' (unknown location)

Can you please help me to fix this? Can you pass me the exact code for this?

[zhaoyiran924]

Thanks for your interest. As you can see from the code, there are two folders in the 'neuron_detection/transformers': 'models' and 'generations', just replacing the corresponding folder in your original transformer package will be fine.

[RadiantCrystal]

Hi @zhaoyiran924, Thanks for your response. Let me try this out once.

[RadiantCrystal]

@zhaoyiran924 , Unfortunately now this kind of issues are coming.

`Traceback (most recent call last):
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1793, in _get_module
return importlib.import_module("." + module_name, self.name)
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/generation/utils.py", line 27, in
from ..deepspeed import is_deepspeed_zero3_enabled
ModuleNotFoundError: No module named 'transformers.deepspeed'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1793, in _get_module
return importlib.import_module("." + module_name, self.name)
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/models/auto/modeling_auto.py", line 21, in
from .auto_factory import (
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 40, in
from ...generation import GenerationMixin
File "", line 1055, in _handle_fromlist
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1781, in getattr
module = self._get_module(self._class_to_module[name])
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1795, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
No module named 'transformers.deepspeed'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/media/rimi/DATA/Rima/multilingual_analysis/neuron_detection/neuron_detection.py", line 17, in
from transformers import AutoModelForCausalLM, AutoTokenizer
File "", line 1055, in _handle_fromlist
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1782, in getattr
value = getattr(module, name)
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1781, in getattr
module = self._get_module(self._class_to_module[name])
File "/home/rimi/anaconda3/envs/SeaExam/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1795, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.auto.modeling_auto because of the following error (look up to see its traceback):
Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
No module named 'transformers.deepspeed'`

[zhaoyiran924]

sry, seems the bug is "No module named 'transformers.deepspeed'"? But I do not use transformers.deepspeed, so I did not encounter that bug. I don't know what will change if implemented on deepspeed.

[RadiantCrystal]

Hi @zhaoyiran924 , I have managed to fix the transformers issue. Although I am now getting a new issue. Could you please help me to fix it? Which part of the code I have to modify?

[zhaoyiran924]

Could you comment the try: and except to check where does the bug come from?

[RadiantCrystal]

Also @zhaoyiran924 , After execution of code python neuron_detection.py english 1000 , I am getting this error.

[RadiantCrystal]

Could you comment the try: and except to check where does the bug come from?

Could you please tell me which line of the code file neuron_detection.py you are referring to? @zhaoyiran924

[zhaoyiran924]

Oh, yes, because the code has bugs, so no neurons are detected. Could you comment the try: and except: to check where is the bug?

[zhaoyiran924]

https://github.com/DAMO-NLP-SG/multilingual_analysis/blob/main/neuron_detection/neuron_detection.py#L73

[RadiantCrystal]

https://github.com/DAMO-NLP-SG/multilingual_analysis/blob/main/neuron_detection/neuron_detection.py#L73

@zhaoyiran924 , Yes. removed try-except. Now this error is coming. Can you please tell me at which line of each code file I need to change what if you consider llama3 as the chosen model?

[zhaoyiran924]

Seems like this line fails to work? https://github.com/DAMO-NLP-SG/multilingual_analysis/blob/main/neuron_detection/transformers/generation/utils.py#L2256
As the early_exit_layers is None now.

[RadiantCrystal]

Seems like this line fails to work? https://github.com/DAMO-NLP-SG/multilingual_analysis/blob/main/neuron_detection/transformers/generation/utils.py#L2256 As the early_exit_layers is None now.

Can you please tell me exactly where I need to set it in the utils.py? In the neuron_detection.py file, candidate_premature_layers is already set. @zhaoyiran924

[zhaoyiran924]

Oh, I think it's because you do not use greedy_search when generating, you need to set the generation_config as greedy decoding. For example, for llama3, you can set as this:

{
"bos_token_id": 128000,
"eos_token_id": 128001,
"transformers_version": "4.40.0.dev0"
}

Or you can set in the model.generate() function here https://github.com/DAMO-NLP-SG/multilingual_analysis/blob/main/neuron_detection/neuron_detection.py#L35,

Sorry I forgot to mention this in the readme

[RadiantCrystal]

Oh, I think it's because you do not use greedy_search when generating, you need to set the generation_config as greedy decoding. For example, for llama3, you can set as this:

{ "bos_token_id": 128000, "eos_token_id": 128001, "transformers_version": "4.40.0.dev0" }

Or you can set in the model.generate() function here https://github.com/DAMO-NLP-SG/multilingual_analysis/blob/main/neuron_detection/neuron_detection.py#L35,

Sorry I forgot to mention this in the readme

Hi @zhaoyiran924 , sorry for asking too many questions. Thanks for continuously answering my queries.

I tried this workaround just now but unfortunately same error is coming up.

Added like this -->

activate_keys_v, activate_keys_o, layer_keys = model.generate(**{'input_ids':inputs.input_ids, 'max_new_tokens':1, 'candidate_premature_layers':candidate_premature_layers,
"bos_token_id": 128000,
"eos_token_id": 128001
})