Hi,
I've been wanting to test LangRank specifically to see how adding some of my own language distances as predictors would fare, but running your script yielded significantly different results from the ones reported in the paper.
This is what I get using your unaltered script (i.e., without my predictors), running calculate_distances.py and replace_distances.py followed by the respective models:
EL (all): 48.9
EL (lang): 53.6
MT (all): 48.0
MT (lang): 33.1
DEP (all): 64.3
DEP (lang): 74.0
POS (all): 22.8
POS (lang): 7.0
I've tried reinstalling packages to match the versions detailed in urielplus-requirements.txt in case there were changes to the imputation method, etc., and this did in fact bring the results closer to the ones reported (although the fancyimpute package remained at the same version):
EL (all): 65.1
EL (lang): 65.9
MT (all): 56.8
MT (lang): 34.9
DEP (all): 75.6
DEP (lang): 73.5
POS (all): 24.2
POS (lang): 13.3
Can you confirm whether you get the results given in the paper using this script and/or whether you have any idea what could have gone wrong?
Cheers,
Tim
Hi,
I've been wanting to test LangRank specifically to see how adding some of my own language distances as predictors would fare, but running your script yielded significantly different results from the ones reported in the paper.
This is what I get using your unaltered script (i.e., without my predictors), running
calculate_distances.pyandreplace_distances.pyfollowed by the respective models:EL (all): 48.9
EL (lang): 53.6
MT (all): 48.0
MT (lang): 33.1
DEP (all): 64.3
DEP (lang): 74.0
POS (all): 22.8
POS (lang): 7.0
I've tried reinstalling packages to match the versions detailed in urielplus-requirements.txt in case there were changes to the imputation method, etc., and this did in fact bring the results closer to the ones reported (although the
fancyimputepackage remained at the same version):EL (all): 65.1
EL (lang): 65.9
MT (all): 56.8
MT (lang): 34.9
DEP (all): 75.6
DEP (lang): 73.5
POS (all): 24.2
POS (lang): 13.3
Can you confirm whether you get the results given in the paper using this script and/or whether you have any idea what could have gone wrong?
Cheers,
Tim