Hello everyone,
I am using the eMFD to analyze some tweets.
When I ran "%%bash
emfdscore sampled_tweets_all_without.csv all-sent.csv bow emfd all sentiment" in jupyter notebook, I obtained
"Traceback (most recent call last):
File "C:\Users\authuser.PITT.conda\envs\emfd\Scripts\emfdscore", line 67, in
df = score_docs(csv,DICT_TYPE,PROB_MAP,SCORE_METHOD,OUT_METRICS,num_docs)
File "C:\Users\authuser.PITT.conda\envs\emfd\lib\site-packages\emfdscore\scoring.py", line 359, in score_docs
scored_docs.append(nlp(row))
File "C:\Users\authuser.PITT\AppData\Roaming\Python\Python39\site-packages\spacy\language.py", line 1059, in call
raise ValueError(Errors.E005.format(name=name, returned_type=type(doc)))
ValueError: [E005] Pipeline component 'mfd_tokenizer' returned <class 'list'> instead of a Doc. If you're using a custom component, maybe you forgot to return the processed Doc?
CalledProcessError Traceback (most recent call last)
Cell In[29], line 1
----> 1 get_ipython().run_cell_magic('bash', '', 'emfdscore sampled_tweets_all_without.csv all-sent.csv bow emfd all sentiment\n')
File ~.conda\envs\emfd\lib\site-packages\IPython\core\interactiveshell.py:2517, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2515 with self.builtin_trap:
2516 args = (magic_arg_s, cell)
-> 2517 result = fn(*args, **kwargs)
2519 # The code below prevents the output from being displayed
2520 # when using magics with decorator @output_can_be_silenced
2521 # when the last Python token in the expression is a ';'.
2522 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):
File ~.conda\envs\emfd\lib\site-packages\IPython\core\magics\script.py:154, in ScriptMagics._make_script_magic..named_script_magic(line, cell)
152 else:
153 line = script
--> 154 return self.shebang(line, cell)
File ~.conda\envs\emfd\lib\site-packages\IPython\core\magics\script.py:314, in ScriptMagics.shebang(self, line, cell)
309 if args.raise_error and p.returncode != 0:
310 # If we get here and p.returncode is still None, we must have
311 # killed it but not yet seen its return code. We don't wait for it,
312 # in case it's stuck in uninterruptible sleep. -9 = SIGKILL
313 rc = p.returncode or -9
--> 314 raise CalledProcessError(rc, cell)
CalledProcessError: Command 'b'emfdscore sampled_tweets_all_without.csv all-sent.csv bow emfd all sentiment\n'' returned non-zero exit status 1."
To deal with this problem, I have tried the following:
Check the mfd_tokenizer function in the emfdscore package to ensure it is returning a SpaCy Doc object. Modify the function if necessary to ensure compliance with SpaCy's pipeline requirements.
Check if there's an updated version of the emfdscore package that resolves this issue. Update the package using: bash
pip install --upgrade emfdscore
Add debugging statements inside the scoring.py file (if editable) to inspect the output of the mfd_tokenizer function. Ensure it transforms input correctly into a Doc object.
My system already has Git Bash, and I ensure it has the necessary permissions and that the paths are set correctly. Use Git Bash to directly invoke the script: bash
"C:/Program Files/Git/bin/bash.exe" -c "python path/to/emfdscore.py args"
Check if the sampled_tweets_all_without.csv file is properly formatted and readable by emfdscore. Ensure there are no invalid characters or unexpected formats that might cause issues during processing.
I override the mfd_tokenizer in the pipeline with a custom tokenizer that correctly returns a Doc object. More specifically, I use the codes like:
import spacy
from spacy.tokens import Doc
from spacy.language import Language
@Language.component("mfd_tokenizer")
def mfd_tokenizer(doc):
tokens = [token.text for token in doc]
return Doc(doc.vocab, words=tokens)
nlp = spacy.load("en_core_web_sm", disable=['ner', 'parser'])
nlp.add_pipe("mfd_tokenizer", first=True)
and
import spacy
def custom_tokenizer(doc):
return nlp.make_doc(doc.text)
nlp = spacy.load("en_core_web_sm")
nlp.replace_pipe("mfd_tokenizer", custom_tokenizer)
Inspect Input and Component Behavior: Run the component on a small sample dataset to isolate the error:
For instance, I try the codes:
from emfdscore.scoring import mfd_tokenizer
doc = nlp("Sample text for testing.")
print(mfd_tokenizer(doc))
The outcome is
ImportError Traceback (most recent call last)
Cell In[26], line 1
----> 1 from emfdscore.scoring import mfd_tokenizer
2 doc = nlp("Sample text for testing.")
3 print(mfd_tokenizer(doc))
ImportError: cannot import name 'mfd_tokenizer' from 'emfdscore.scoring' (C:\Users\authuser.PITT.conda\envs\emfd\lib\site-packages\emfdscore\scoring.py)
Installation of Spacy 3.4.0 provided by @uless
All of these methods have failed. Hence, could you please tell me how I can deal with this problem?
Thanks very much for all your help!
Hello everyone,
I am using the eMFD to analyze some tweets.
When I ran "%%bash
emfdscore sampled_tweets_all_without.csv all-sent.csv bow emfd all sentiment" in jupyter notebook, I obtained
"Traceback (most recent call last):
File "C:\Users\authuser.PITT.conda\envs\emfd\Scripts\emfdscore", line 67, in
df = score_docs(csv,DICT_TYPE,PROB_MAP,SCORE_METHOD,OUT_METRICS,num_docs)
File "C:\Users\authuser.PITT.conda\envs\emfd\lib\site-packages\emfdscore\scoring.py", line 359, in score_docs
scored_docs.append(nlp(row))
File "C:\Users\authuser.PITT\AppData\Roaming\Python\Python39\site-packages\spacy\language.py", line 1059, in call
raise ValueError(Errors.E005.format(name=name, returned_type=type(doc)))
ValueError: [E005] Pipeline component 'mfd_tokenizer' returned <class 'list'> instead of a Doc. If you're using a custom component, maybe you forgot to return the processed Doc?
CalledProcessError Traceback (most recent call last)
Cell In[29], line 1
----> 1 get_ipython().run_cell_magic('bash', '', 'emfdscore sampled_tweets_all_without.csv all-sent.csv bow emfd all sentiment\n')
File ~.conda\envs\emfd\lib\site-packages\IPython\core\interactiveshell.py:2517, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2515 with self.builtin_trap:
2516 args = (magic_arg_s, cell)
-> 2517 result = fn(*args, **kwargs)
2519 # The code below prevents the output from being displayed
2520 # when using magics with decorator @output_can_be_silenced
2521 # when the last Python token in the expression is a ';'.
2522 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):
File ~.conda\envs\emfd\lib\site-packages\IPython\core\magics\script.py:154, in ScriptMagics._make_script_magic..named_script_magic(line, cell)
152 else:
153 line = script
--> 154 return self.shebang(line, cell)
File ~.conda\envs\emfd\lib\site-packages\IPython\core\magics\script.py:314, in ScriptMagics.shebang(self, line, cell)
309 if args.raise_error and p.returncode != 0:
310 # If we get here and p.returncode is still None, we must have
311 # killed it but not yet seen its return code. We don't wait for it,
312 # in case it's stuck in uninterruptible sleep. -9 = SIGKILL
313 rc = p.returncode or -9
--> 314 raise CalledProcessError(rc, cell)
CalledProcessError: Command 'b'emfdscore sampled_tweets_all_without.csv all-sent.csv bow emfd all sentiment\n'' returned non-zero exit status 1."
To deal with this problem, I have tried the following:
Check the mfd_tokenizer function in the emfdscore package to ensure it is returning a SpaCy Doc object. Modify the function if necessary to ensure compliance with SpaCy's pipeline requirements.
Check if there's an updated version of the emfdscore package that resolves this issue. Update the package using: bash
pip install --upgrade emfdscore
Add debugging statements inside the scoring.py file (if editable) to inspect the output of the mfd_tokenizer function. Ensure it transforms input correctly into a Doc object.
My system already has Git Bash, and I ensure it has the necessary permissions and that the paths are set correctly. Use Git Bash to directly invoke the script: bash
"C:/Program Files/Git/bin/bash.exe" -c "python path/to/emfdscore.py args"
Check if the sampled_tweets_all_without.csv file is properly formatted and readable by emfdscore. Ensure there are no invalid characters or unexpected formats that might cause issues during processing.
I override the mfd_tokenizer in the pipeline with a custom tokenizer that correctly returns a Doc object. More specifically, I use the codes like:
import spacy
from spacy.tokens import Doc
from spacy.language import Language
@Language.component("mfd_tokenizer")
def mfd_tokenizer(doc):
tokens = [token.text for token in doc]
return Doc(doc.vocab, words=tokens)
nlp = spacy.load("en_core_web_sm", disable=['ner', 'parser'])
nlp.add_pipe("mfd_tokenizer", first=True)
and
import spacy
def custom_tokenizer(doc):
return nlp.make_doc(doc.text)
nlp = spacy.load("en_core_web_sm")
nlp.replace_pipe("mfd_tokenizer", custom_tokenizer)
Inspect Input and Component Behavior: Run the component on a small sample dataset to isolate the error:
For instance, I try the codes:
from emfdscore.scoring import mfd_tokenizer
doc = nlp("Sample text for testing.")
print(mfd_tokenizer(doc))
The outcome is
ImportError Traceback (most recent call last)
Cell In[26], line 1
----> 1 from emfdscore.scoring import mfd_tokenizer
2 doc = nlp("Sample text for testing.")
3 print(mfd_tokenizer(doc))
ImportError: cannot import name 'mfd_tokenizer' from 'emfdscore.scoring' (C:\Users\authuser.PITT.conda\envs\emfd\lib\site-packages\emfdscore\scoring.py)
Installation of Spacy 3.4.0 provided by @uless
All of these methods have failed. Hence, could you please tell me how I can deal with this problem?
Thanks very much for all your help!