rustysec

Gold Pressed Latinum

Russ Morris

A year ago I would never have guessed that I would be documenting my homelab adventures in hosting AI models. As it turns out, I stumbled into this hobby by accident and became increasingly fascinated the more I experimented.

As I was transitioning out of my last corporate role, upper management was aggressively pushing the use of coding assistants. My team was curious but not entirely confident in the efficacy or privacy guarantees of the products at the time. While the newness of the tech was interesting and appealing, it was hard to tell whether AI was mostly hype or if it would be truly transformative, like LSP (Language Server Protocol) a decade or so before. After resigning from that job, I began experimenting quite a bit with smaller models on my personal M4 MacBook Pro and later acquired a new Zephyrus G14 to run Linux.

Initial Setup

Neither of these systems was quite powerful enough or had enough gold pressed latinum RAM/VRAM to really run larger models effectively. So I did the only reasonable thing I could think of and experimented with llama.cpp + RPC (Remote Procedure Call) + Ethernet over Thunderbolt 3 (TB3). While using Ethernet + TB3 is better than RPC over Wi-Fi or 1 Gb/s Ethernet, it is much worse than something like RDMA (Remote Direct Memory Access) over Thunderbolt or PCIe.

Machine Specs:

  • MBP: M4 24 GB Unified RAM
  • G14: 32 GB RAM, RTX 5070 Ti Mobile (12 GB VRAM), Fedora 44 (and CachyOS)

With this setup I was able to achieve somewhere in the 25-35 tokens/sec (t/s) range running Q4/Q5 of Qwen 3.5 or Qwen 3.6 via RPC and MTP (multi-token prediction), which was pretty great, all things considered. This, however, left me at the slight disadvantage of needing both systems side by side and connected with a USB-C cable to run these larger quantizations. As you might guess, this is a terrible experience if you are ever "on the go".

Next Iteration

This eventually led me to sell my G14 and acquire a 128 GB Strix Halo machine in the form of a Framework Desktop Mainboard. Now, these things were fairly pricey before RAMmageddon (the RAM price spike) and all the Strix boxes are simply absurd now. I slapped the mainboard into a DeskPi RackMate T0 using Framework's rack tray, which all worked out perfectly.

Once the new NVMe and wireless card arrived and everything was assembled, I fired the whole rig up and installed Fedora 44. I build llama.cpp from source every couple of days, compiling specifically for Vulkan.

cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF &&
cmake --build build --config Release -j

The ROCm/HIP vs Vulkan situation is fairly interesting, with ROCm "usually" winning on prefill but Vulkan "usually" winning on decode. Given that these generalities often switch places depending on the model architecture and llama.cpp support maturity, I have found that Vulkan has the best overall performance on the Strix Halo platform.

I also run Open WebUI to provide a user-friendly chat interface for myself and my family. The llama.cpp instances are managed by Llama Swap, which does a great job of letting you configure which models can run side-by-side on your host.

models:
  # Llama Swap lets you define any number of models
  # you want to have access to by using llama-server
  # or vllm
  "Qwen3.8-Flash":
    cmd: |
      ${q38-llama}
      --model ${models-dir}/qwen3.8-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
      --mmproj ${models-dir}/qwen3.8-flash-next/mmproj-F16.gguf
      --model-draft ${models-dir}/qwen3.8-flash-next/mtp/mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf
      --spec-draft-n-max 3
      --spec-type draft-mtp
      --spec-draft-p-min 0.75
      --ctx-size 262144
      --flash-attn on
      --n-gpu-layers 999
      --n-gpu-layers-draft 999
      --temp 1.0
      --top-p 0.95
      --min-p 0.0
      --top-k 20
      --presence-penalty 0.0
      --repeat-penalty 1.0
      --parallel 1
      --reasoning-preserve
      --fit off
      --image-min-tokens 1024
      --kv-unified

  # Other models here...
  # DeepSeek-v4-Flash:
  # ...
  # ...

# You then configure the combinations in which you
# can run the models. This is incredibly powerful,
# the new "matrix" feature allows you to mix and match,
# but you are responsible for resolving the memory conflicts.
# Example: Don't allow DeepSeek and Qwen 3.8 Flash Next to
# load at the same time: you will reach OOM (out of memory)!
routing:
  router:
    use: matrix
    settings:
      matrix:
        vars:
          g4: Gemma4
          q36: Qwen3.6
          q36it: Qwen3.6-it
          q38: Qwen3.8
          q38it: Qwen3.8-it
          lag21: Laguna-S-2.1
          ds4f: DeepSeek-v4-Flash
          q38f: Qwen3.8-Flash
          l30f: Ling-3.0-Flash

        sets:
          # allows two models from the superset to be loaded together
          qwen: "(q36 | q36it | q38 | q38it) & (q36 | q36it | q38 | q38it)"
          # gemma + one smaller qwen  model
          gemma: "g4 & (q36 | q36it | q38 | q38it)"
          # only one of the larger models
          code: "ds4f | q38f | l30f | lag21"

In front of all this is nginx, which provides TLS for Open WebUI.

The final piece of the puzzle is remote access when I'm not at home, which I solve through Tailscale. All my machines and phones run the Tailscale client so they are meshed no matter where I am.

Performance

While I consider this setup to be "performant enough for me", it will never match the frontier API performance. These are some very rough numbers:

Note: The version of Ling 3.0 that I run was quantized locally using the standard llama.cpp tools.

Some rough frontier API numbers for comparison (at the time of writing):

  • GPT 5.3 Spark: 1000 t/s
  • GPT 5.3: ~100 t/s
  • Claude Haiku & Sonnet: ~80 t/s
  • DeepSeek v4: 50+ t/s
  • DeepSeek v4 Pro: 150+ t/s
  • GLM 5.3 Flash: 250+ t/s

It's apparent from these numbers that paid API access is faster, but what the numbers don't show is that it's also, generally, more intelligent. A quick lesson in quantization: the frontier/API access models are usually running at 16-bit (BF16/FP8) precision. Quantization is a form of "compressing" the model down so that the "knowledge" is instead represented by a narrower bit width. For example, Q4 is 4 bits instead of 16. If you compare the output quality of a full-precision model with that of a Q8 (half the precision), the deviation is almost imperceptible. However, when you start moving below Q8, you start to see degradation that can be measured: Q6 claims to be "nearly lossless", Q5 "almost lossless", and Q4 is where most users start to notice a quality drop-off. You can read more in Unsloth's dynamic-3.0 GGUF guide, where the team has done a remarkable job of breaking down the technology at various levels. The version of DeepSeek v4 Flash I am able to host is at a UD-IQ3_XXS quantization, while the hosted version is able to run at full precision. Qwen 3.8 Flash Next is a similar story: I am only able to fit Q4 in my system's unified RAM.

There is a caveat to the quantization precision: a model with more parameters can often survive quantization well enough to end up still being "better" than a smaller model (meaning fewer parameters) at a higher precision. This leads to a lot of disagreement and "trust me" benchmarking regarding which models are best at a particular memory footprint. A quick example from above: I am running Ling 3.0 Flash at Q6, which maintains a higher precision, but it contains "only" 127B parameters while DeepSeek v4 Flash contains 304B at UD-IQ3.

Possible Next Steps

AMD is about to release "Gorgon Halo", which will have improved GPU performance and 192 GB of gold pressed latinum unified RAM/VRAM. There is the possibility that this would either provide a nice additional system to cluster with my current machine, or even drive down the price of Strix Halo machines to make doubling up on them more financially viable. I don't have great confidence in prices becoming any more "consumer friendly", but a guy can hope.

Notes On Price

There isn't really a way to justify this setup as a cost-savings measure or as a "smart investment" when considering how inexpensive API access to adequate models like DeepSeek is. The entire rig I built cost ~$4000, which is a lot. Qwen 3.8 Flash on OpenRouter costs $0.04 per million input tokens and $0.47 per million output tokens. It would take something like 16,000,000,000 combined input and output tokens to "make up" the cost difference. This entire adventure was about more than just price: I really value data sovereignty and privacy, and I just don't have a lot of trust in the current product landscape to hand over more of my family's data unless it is necessary.

Edit 2026-10-05: I've been using some frontier lab products recently and am actively re-evaluating my 'trust level' of the available offerings. The entire landscape is constantly evolving and I plan to evolve with it.

OK, But Why?

Like many people, my initial use of all things "AI" was software development. After all this, I mostly just want to wire pi to a competent model like DeepSeek v4 Flash or Qwen 3.8 Flash Next. This setup absolutely lets me do that from wherever I am in the world, which is very exciting.

Additionally, I enjoy an environment where I can experiment with tools and development without worrying about token limits. It's fun to create some MCP servers that fit my perfect needs and allow these models to run wild with them. The curious hacker kid in me simply loves to tinker and bend technology to my own needs, and this is an excellent way to flex those muscles.

And finally, being able to control and monitor the technology my kids have access to is very important to our family. While they currently don't have much contact with AI products, they will eventually and I want to be equipped to shepherd them along that journey.

Wrap Up

I have a firm belief that these AI models and the associated tools will be part of software development going forward. Getting my hands dirty hosting these models has given me a clearer sense of their strengths and weaknesses, which ultimately has helped me adjust my expectations and appreciation for them as a tool. While I do not think vibe coding will replace software engineering as a whole, I do encourage developers to at least make contact with the technology to understand it better. See if you can use it in a way that enhances your workflow or frees up some of your human cycles for different work.

About a decade ago, LSP was introduced as a standard, and a lot of developers I knew were very negative about the idea. I heard a lot of "that's lazy" or "but I know the APIs in my head!" That may all be true, but I also know that LSPs are a foundational part of modern IDEs and developer workflows. I see AI harnesses and tools in much the same way: they lift a lot of burdens off developers who can then choose to be lazy, or choose to apply their organic brain cycles to interesting problems while the robots help with everything else.

I don't "know" this is the right stance. However, I have taken a very measured approach to the technology over the past few years, and it seems like my evaluations have been mostly spot-on every step of the way.

Or it's just confirmation bias.

One of the two.