Skip to content

Evaluator omits an output with the empty inputs #1032

@de9uch1

Description

@de9uch1

Bug description

Evaluator class should return the N scores for N inputs, but it omits an output when an empty input is given.

How to reproduce

  1. Prepare the python environment
  2. pip install pymarian
  3. Run the following codes:
from huggingface_hub import hf_hub_download as hf_get
from pymarian import Evaluator

model_id = "marian-nmt/bleurt-20"
model = hf_get(model_id, filename="checkpoints/marian.model.bin")
vocab = hf_get(model_id, filename="vocab.spm")

evaluator = Evaluator.new(model_file=model, vocab_file=vocab, like="bleurt", cpu_threads=8)

for score in evaluator.evaluate(["\t"]):
    print(score)

It causes the assertion error:

Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "<path_to_pymarian>/pymarian/__init__.py", line 142, in evaluate
    assert len(scores) == len(batch)
AssertionError

In this example, the len(batch) will be 1, but len(scores) will be 0.

Context

  • Python version: Python 3.10.15
  • Marian version: v1.12.31 dd3571de

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions