IT LIVES

[CoolJohn-lab]

Introducing my just-completed Odysseus self-building workspace setup. I have spent the last month or so fighting with graphics cards and Linux and vLLM and god knows what else but now it is WORKING. and I don't mean just working I mean WORKING. This thing is doing real, actual work for me, and is much faster and in some ways BETTER than my usual workhorse, Grok 4.5 (Actually 4.6 as of this week lol).

What I have:

4x 5060 Ti 16GB for a total of 64GB VRAM
an old 4 slot motherboard from 2014
an old xeon
32GB of DDR4 ram
an old 500GB nVME drive I had spare
Ubuntu Server
vLLM (my own highly tuned serve, not cookbook defaults. you must tune vLLM for your specific hardware)
Odysseus (my fork)

What that gets me:

Qwen 3.6 35B at FP8 precision
Vision enabled
MPS level 3 enabled
~750K total context window.
Around 180 Tok/sec. Yes I'm not joking. that's average as well. it spikes a lot higher than that.

Qwen caps out at 256k so that means I can run 3 sessions concurrently with no danger of OOM (4 if I want to be brave).

This thing is pretty cutting edge although I am skipping the newer models. I have tried MANY MODELS optimal for 64GB and nothing beats Qwen. Ornith comes close but Qwen is about two times faster and has vision. I use Ornith for heavy coding workloads as it is really good at terminal and coding, better than Qwen, but it's not really worth the 5-6 minutes it takes to swap most of the time, Qwen codes like an absolute champ. The only thing that will ever unseat Qwen 3.6 35B FP8 from this machine will be Qwen 3.7, which is coming out... tomorrow hahahaha this hobby is crazy. something new and groundbreaking every few days. I am literally losing my mind.

What the heck am I doing in this screenshot:

I feel so crazy, like dexter's lab or something. What I'm doing here is this. I have VS Code Insisders build open on the left hand side of the screen, and it is hooked up to my vLLM and using Qwen as its Github Copilot agent. I only have a free account on VS Code and that is fine for you to add custom models and use them exclusively, for now anyway...

Then, inside VS code I have asked qwen to open an Odysseus window, which is has done. The funky multicoloured rainbow around the outside of the Odysseus frame is VS code telling me that an agent has full control over that window and can read it. I think it looks incredible, and I didn't even know VS Code had this feature, the agent just started using it on its own! I've done a bit of customisation to get this working well, but not much, this is mainly just VS Code's (Insider's build) default agent harness. I used this harness about 2 months ago and it was garbage. What it is now, as of about a week ago (!!!) as a harness it beats even Cursor or Claude Code or Codex, all of them. It's phenomenal, it's doing stuff I didn't even know was possible. VS Code Insiders built ftw.

So yeah I have VS Code set up with skills and whatnot to be able to invoke Odysseus agents. It doesn't have to open a window like this and control it, but it looks pretty damn cool if it does so this is the default way I have it spin up Oddy sessions :D in some ways this is Qwen interacting with itself, but via two different harnesses. This is mind bending to me. This hobby will take your sanity, smash it up and feed it to you as tokens. I can ask VS code to improve Odysseus or add a feature, deploy the feature, restart Odysseus, and test its own changes live in this window. It does a phenomenal job of it too. I had it rip out the existing memories feature and replace it with a custom vault memory system. I removed "Cookbook" and replaced it with a simple model switcher menu so I can load different models with one click. This thing is BUILDING. ITSELF. and testing itself too.

THEN.

THEN my friends, I have Cursor open on the right hand portion of the screen, with my true one and only, my bae Cursor Grok 4.6 High Effort, loaded and ready to go. Grok is not a cheap date, so I don't use it much any more, I do all my coding and development on VS Code using Qwen or Ornith, and then only if it gets stuck or the task is too hard, I have the exact same repos loaded in Cursor, and I can just ask Grok to take over.

The other thing Grok is pretty great at is evaluating how good my Oddy and VS Code setup is doing, it helps me write the rules and AGENTS.MD for the VS Code agent to follow, it helps me write the Skills and Tools and routines for Oddy and debug and tune them... It's doing all the hard work while I just watch 3 agent sessions at once work by themselves on my screen. 2 self-hosted ones that I run for FREE and have full access to all my personal shit, and one via Cursor that's quite expensive and I use sparingly and carefully, and I don't give it the tool access it would need to get to my personal data. Pretty soon I won't need Grok at all, I don't think.

Most days I can't get my jaw off the floor. I have an agent that reads my Gmail and deletes the spam, another agent that adds appointments to my calendar for me based on emails I get. The dream. The promise. it is real. I've built little projects as well, not just Odysseus building Odysseus, little apps and games and stuff, just because I can. I can't wait to move on to building stuff with this box now that I'm more or less done setting it up. I will have to spend a day or two tuning Qwen 3.8 this weekend, but after that, bring it on! Vibes-based only, but I feel like I have something more-or-less between Sonnet 4.6 and Sonnet 5 in terms of capability, but it runs about 3 times faster than Sonnet does via the API. I get 256k context windows the same as Sonnet does, and I can create Subagents in both VS Code and in Oddy just like Cursor can. My model can reason, control my PC, take screenshots and read them, control my mouse and keyboard, and do pretty much anything the big models can other than generate images.

Thank you Felix for making this possible and inspiring me to build my first ever dedicated inference rig, Fred. he is a BEAST. :D

[CoolJohn-lab]

Ok all I can say is wow. Qwen 3.8 27B FP8 has been running for 24 hours now and OH BOY is it smarter than 3.6! quite phenomenal.

It's much slower than 3.6 35B-A3B, I'm getting less than half the max tok/sec on the same tuned vLLM setup. They really did something there with that A3B model. I hope we one day see an A3B or A5B for 3.8, but for now I am getting around 75 tok/sec on the dense model, after tuning vLLM a little, and I have over 800K of context window to play with so I can safely run 3 agents at once or I can run 4 if I soft-cap them to 200K.

64GB of VRAM really unlocks this model and I can't recommend it on anything lower. 60GB at a push if you're on a Mac. You can get by on Q4 or NVFP4 but it just isn't the same. on FP8 this thing is a demon.

I'm still capped to 262k total context window in 3.8, they sadly weren't able to bring native 1M windows to the 27B it seems. I haven't experimented with YaRN yet, although I plan to, as it's meant to be really good on this.

As a dense model it thinks a lot on complex tasks, for several minutes usually, but it does its thinking with relatively few tokens. I get a short prefill spinup of about 5 seconds, much shorter than the A3B 10-20 seconds, but then a few quick turns to get the lay of the land, followed by very long turn of internal deliberation, which burns maybe 30-40K tokens. But after its done with that it still has over 200k left to make calm, measured actions which generally work first time.

I am finding complex tasks like building a website or refactoring a project are finishing successfully at 120K-150K tokens total, nowhere near the limit, then I can spend the remaining ~100K on updating documentation and testing, with all the plan and the build still in the window so it still knows everything that it did. It's pretty phenomenal. Getting a full plan, feature build, test and documentation pass done in ~200K tokens, FOR FREE, ON MY OWN HARDWARE, feels like witchcraft, quite frankly.

Yeah, it's slower per turn than the 3.6 A3B, much slower, it thinks for literally 10 minutes at a time, but it's using WAY less turns because its getting things right first time far more often. Whereas with the 3.6 35B-A3B what you got was maybe 10 seconds of thought then it launched right into the task, got it wrong 10 times, self-corrected 10 times, and came out the other side in the same place, but actually all the rework takes longer overall than just getting it right first time, slowly.

Mistakes burn through tokens: Debugging, fixing, redeploying, "did the fix work?", "Yes"... all of that is tonnes of useless tokens that don't get you anything, and bloat your window so you can't do proper documentation after you're done. So it is well worth spending a decent amount up of time and tokens up front., to prevent bugs in the first place, which is exactly what this model does. It really is the tortoise and the hare time again.