Model inject via tritonclient API

[sapjunior]

It would be beneficial if we there is a way to inject/passthrough model binary via API rather than current single way (via model file path). For example, in on premise use case which require model encryption/protection, user can writing their own mechanism to protect model and inject model into triton and unload without using filesystem as a medium

[the-david-oy]

@GuanLuo @tanmayv25 Do you know if this is somehow doable? Could we just have the model frameworks load the binaries directly at runtime?

[tanmayv25]

Yes. It is already supported. See the documentation on this feature here: https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_model_repository.md#load

[tanmayv25]

If using the python clients then look at the config and file options: https://github.com/triton-inference-server/client/blob/main/src/python/library/tritonclient/grpc/__init__.py#L656