Improvements inspired by GLUE

[piotrpolec]

Following meetings we had about GLUE/Superglue, there are some improvements we could make to SPEECHM

1. Group submission
   Since glue is a static challenge (speaking in our terms) and is not as heavily configurable as in our case, theirs submission works much more straightforward - user simply submits a zip file with all hypotheses. We could implement it and I propose the following structure:

• inside a zip file there is a folder for each task user wants to submit.
• inside each task folder, there are folders named after each test_set
• inside test_set folder, there is a file named from_language_to_language that is a hypothesis user wants to submit
  Alternatively, we could skip test set folders and include their name in the hypothesis, I don't mind. I think it all depends on how many test sets have multiple entries. In SPEECHM case, many have only one entry, so extra folder would be more of a hassle

At some point, we will need to validate the input - but to do so, I would either introduce it in single upload or introduce the group submission without the validation.

2. Introduce submission limits
   Glue has submission limit at two per day, six per month and it's something we could also do

3. Require paper submission when submitting a model
   This is an extra form of validation, that a user doesn't upload something faulty

[mkasztelnik]

We can also provide the option to download a zip file with the structure created (based on the model configuration - assignment to tasks), where hypotheses are represented by empty files that should be populated.

[piotrpolec]

With that assumption we'd have to additionally allow for empty folders in case user wants to participate only in certain languages (they'll be present from the template, but not have anything in them), but it's doable, nice idea. I suggest adding this as a separate PR. In the first one we'll add uploading, in a separate one, constructing the structured folder based on the model
Do you think it should be only folders or template files for each of the test set entries too?

[piotrpolec]

I also had the idea that the upload itself will create a new submission - user then would define which tasks/models they want to participate in by defining the folders inside - that limits us from having the template on the model level, but we could provide a template on the task level.
This way, we won't have to validate if model config is consistent with the upload config. We would only have to check if it's consistent with tasks/test sets defined in the challenge
WDYT? @mkasztelnik

[mkasztelnik]

This can be the next step after we validate the zip upload feature.

[piotrpolec]

decision whether to create model before/during upload is something we need to decide right now, since hypothesis has to be a part of a model

[piotrpolec]

Regarding rate limiting the submissions, it's more complicated than in GLUE. SPEECHM allows to upload a singular hypothesis and will allow group upload. We also allow to delete the hypothesis for reupload.

1. We could simply limit the number of group submissions, but user would still be able to upload singular hypothesis.
2. We could also count the number needed files for the model and allow creating up to said limit of hypothesis times eg. 4 per month to simulate 4 full group submissions. That solution would also limit uploading singular hypothesis, but is more complex.

Sometimes user can submit something that throws an error, should that count towards the submission limit?

There's also a question whether we need to rate limit submissions at all - we do not perform when user submits them, only managers can submit them.

[piotrpolec]

group submission and group submission templates have been introduced
Rate limit and including paper inside of the submission is left to discuss