Dumb Crashing

[ulysses2026-terra]

I am experiencing Dumb Crashing. I got through the process of downloading an LLM. For example maybe I download DeepSeek-R1-0528-Qwen-8B and run this. I am running on a mini PC with no GPU other than integrated, but a large 96 GB of available RAM.

What I am experiencing attempting to just chat with the models (like a local replacement for ChatGPT) is that the chat cuts out and reports that the endpoint is offline, and then the Cookbook shows a red icon but says "ready" and doesn't actually know that the model is offline or what's happening, and sometimes restarting it will let me do a few more queries.

This a terrible state of affairs. I'm trying to do the most basic task of just chatting with an LLM using this -- the point being that it's offline and free of big tech in that way. And yet, in an offline, closed-loop situation that should be capable of infinitely running as long as my computer has electricity and available RAM resources, instead the LLM server part crashes randomly and there doesn't seem to be a straightforward crash report that would say what's even happening.

Am I the only one experiencing this? What's the suggested solution?

[ulysses2026-terra]

Perhaps the reason this is most frustrating to me is that it worked to begin with. I can submit 2 or 3 prompts to a model and then it'll die and nothing works, and using Restart in cookbook on the serve job will allegedly restart it but it still won't chat properly in the Chat section.

This feels as if the project is being sabotaged by OpenAI or Google or Microsoft or some other big tech bad actor, who would very easily have the resources to submit bad commits to this project laced with the appearance of good intentions. Am I wrong?

[ulysses2026-terra]

"Endpoint offline - canceling in 5s" is extremely annoying when doing local LLM operations in the middle of a conversation that was already working. (I'm doing this with no wifi and ethernet cable unplugged -- it's an offline machine!) It feels like being censored or blocked from your own hardware, because it was already working previously. It's like talking to your pet rock only to have someone take it away.

[ulysses2026-terra]

This really makes the project feel like absolute vibe-coded hogwash garbage. In particular because I am targeting a very minimal set of features -- I just want to chat with an LLM, and it was already observed to be working and responding to my chats. But now it's not so I'm here complaining on the GitHub instead, in the hopes someone will provide me with some idea about how to debug or find a technical detail that explains why the other party would drop out in the middle of a conversation when the other part is literally my local hardware with no other programs running.

[CoolJohn-lab]

Your computer is too weak to run local models effectively, sorry. Pure CPU inferencing will be so slow that tools like this just time out and assume the endpoint is offline.

You need a GPU or two my friend.

[ulysses2026-terra]

What is the reason that the local model ran at something like 10 tokens per second on the screen some of the time, before the crashing? That is the part I don't understand. If it had never worked at all, I would totally appreciate what you're saying. But it's very confusing that it did work, then stopped working.

[CoolJohn-lab]

Running your own LLM is incredibly confusing, I don’t blame you. They aren’t like normal computer programs, they fail in really weird and unpredictable ways. Sometimes it just do be like that. The thing runs out of context space and brain farts, or internally times out and just stops.