Batch inference

[Dadoou]

Hi, nice work.

Does the model support batch inference?
Batch encode or batch decode
If supported, how to padding the input to make batch?

[shahin-trunk]

+1

[AbrahamSanders]

It doesn't look like batch inference is supported. In theory it could be, but the encode_code function assumes a batch size of 1:

neucodec/neucodec/model.py

Lines 146 to 148 in 2454f32

	semantic_features = self.feature_extractor(
	y.squeeze(0), sampling_rate=16_000, return_tensors="pt"
	).input_features.to(self.device)

Here, y is a tensor of shape [B, 1, T] which is squeezed to [1, T] before passing to the w2v2-bert feature extractor. If B > 1, the feature extractor will throw an exception because it expects batched inputs to be a List[torch.Tensor] rather than a torch.Tensor with a batch dimension.

If we replace y.squeeze(0) with list(y) then the feature extractor successfully returns a tensor of shape [16, 1500, 160] for a y of shape [16, 1, 480320] as expected. However, even if making this change, the rest of the inference chain needs to be tested for correctness with B > 1. Maybe I'll do this in the near future and submit a PR, if I have time.