models being mut (forward)

[jondot]

Hi, I've been studying the code, and wondering about the approach of the models being mut which cause the pipelines to be mut. Looking at that, indeed they're arc/mutex protected, however -- is there any state that lingers between unrelated requests when the server is running?

[EricLBuehler]

Hi @jondot! You are correct that the forward methods of the models do not need to be &mut - there is no shared state across forward passes other than the KV cache, which is managed by the CacheManager and handled safely.

However, Pipeline::step cannot take &self due to NormalModel::activate_adapters/IsqPipelineMixin::re_isq_model and others which take &mut self. Changes made here are persistent across requests because they affect the model itself.

I'll make some changes to remove the unnecessary &muts, thank you for pointing this out. If you have any other questions, please feel free to raise an issue or contact me on our Discord server!

[EricLBuehler]

@jondot I converted this to a discussion to continue our conversation, please feel free to open another issue!