Why is my llm download taking so long and growing larger for a small model

[ps2reimagined]

I wanted to test and experiment with the workspace, so I tried downloading Qwen3.5-4B (Q4-K-M) because from research and looking at huggingface, its a small model (almost 5gb) and can run on my hardware (9070xt gpu, 9800x3d cpu, 16gbs ddr5 ram) according to the built-in calculator. However, after it kept failing, it finally started downloading, but it went on for hours and the size of the download kept growing larger.

I am new to llms and currently learning a bit of coding, so my knowledge is not that great. Could someone help explain why this is happening or if I made a mistake somewhere?

Thanks.

[AnnoDomine]

Hi buddy. No worries, you can cancel the download. The first download actually already fetched the model itself, so theoretically it should work now, assuming it's implemented correctly in Cookbook.
What's happening here is quite simple: You are currently downloading both the base model (which is already finished) as well as all the quantized versions (which are smaller and more performant than the base model).

[ps2reimagined]

Thank you for clarifying. Should I manually delete the other versions from the odysseus folder? It's taking up like 50gbs.

[AnnoDomine]

Sorry, I can't give you a definitive 'yes' here, as I'm personally waiting for the project to become a bit more 'production-ready' before I dive into the usage by myself.

Generally speaking, you can safely remove the files that aren't being used—but as a rule of thumb: If you are unsure which files are needed, create a backup of your data folder first. If the application is designed well, it should only load the model files it's currently configured to use. Check your config or documentation to see which specific quantized versions are actually referenced!

[ps2reimagined]

That's ok and honestly makes sense as this is still very new. So far the delete button within Odyssesus doesn't actually delete anything I've downloaded, everything comes back after a refresh, and the models aren't getting picked up (or maybe I'm making mistakes with the endpoint local host url). I found where the extra quantized versions live within the main oddysseus folder in the huggingface folder as it is 57Gbs and created around the time I began the downlaods, so I might just delete that and hopefully nothing breaks. The best solution I've found so far online is to download or pull models from ollama instead, and connect them that way with an endpoint url.