Returns translated sentences immediately in beam search

[rhenry-nv]

Description

This PR adds a callback mechanism to the Marian beam search to allow sentences to be consumed as soon as they are translated. This PR also updates the triton backend in the Marian contrib folder to leverage this feature.

When running with Triton on batch 64, I observed 1.8X reduction in median latencies and 1.6X reduction in 99.9 percentile latencies for a proxy model. This proxy model is not the same model referred to in prior PRs.

List of changes:

• Adds call back mechanism to beam search
• Adds new ModelTask for Async return
• Updates the Triton backend to leverage the async return feature

Added dependencies: none

How to test

Build a backend by running the build.sh script in contrib/triton-aml. Copy the libtriton_marian.so as described in the README in the contrib/triton-aml folder.

The regression tests should not be affected by this change. I have not checked this, but I have manually checked for correctness both through Triton and without Triton using some sample inputs and outputs.

Describe how you have tested your code, including OS and the cmake command.
cmake .. -DCOMPILE_CPU=on -DCOMPILE_CUDA=on -DUSE_SENTENCEPIECE=on -DUSE_STATIC_LIBS=off -DCOMPILE_SERVER=off -DUSE_FBGEMM=on -DCOMPILE_CUDA_SM35=off -DCOMPILE_CUDA_SM50=off -DCOMPILE_CUDA_SM60=off -DCOMPILE_CUDA_SM70=on -DCOMPILE_CUDA_SM75=off -DCOMPILE_TESTS=on

Ubuntu - 18.04.3 LTS
nvcc - 11.1.105
gcc - 7.5.0

Checklist

• I have tested the code manually
• I have run regression tests
• I have read and followed CONTRIBUTING.md
• I have updated CHANGELOG.md