Post History

Current version by Nick Antonaccio

Current VersionMay 11, 2026 at 00:18

You'd be extremely limited with that machine. Yes, you do need some of that unified memory to run the OS. You'll be stuck running tiny, basically toy models, with very little working space for KV cache. Something like Qwen 3.5 9b will max it out, and don't plan on writing any production code with that model.

Do a Google search, or use ChatGPT to help find machines which will run at least Qwen 3.6 35b and Gemma 4 26b, at least at 4 bit quantization. That's a minimum barrier of entry for getting any actual work completed with self-hosted models. You may still be able to find some sub-$1000 Windows laptops, for example, with a dedicated mobile RTX 3080ti GPU (not the desktop 3080 - the desktop version of that GPU does not have enough VRAM). Those are at the lowest end of usable hardware for any sort of practical LLM inference.

You need a bare minimum of 16 GB VRAM on a dedicated GPU, or at least 32Gb shared RAM on an M series Mac, to do anything useful, but even 32Gb on any of the Macs with unified memory, won't leave you any real room for context, even with heavily compressed models. If you want to stick with Mac, you should really have at least 64Gb unified memory, to get any actual inference work done. You should expect a minimum $1500 for a used Mac Studio (M1 Max) with 64Gb, and more for a MacBook Pro 14" (M1 Max).

Consider looking alternately for a tower with an RTX 3090 or dual RTX 3060s. You can find RTX 3060s all over the place for less than $300 each, and 2 of them give you 24Gb VRAM - that's probably still the best buy on the market for running some of the smallest useful models. If you happen to have a motherboard that can support them, that GPU is a no-brainer for price/performance. You're still going to be really constrained with those small GPUs, but the newest Qwen 3.6 and Gemma 4 Mixture of Expert models can run usably fast on them, and they can get tasks done, especially if you're using a lightweight agentic harness like Pi.

A Strix Halo machine like https://www.amazon.com/gp/product/B0DW238TXK will put you into an entirely different class of LLM inference. It's a world of difference. Only look at the 128Gb shared RAM models, if you want the step up in LLM inference performance (do not get a 32GB model - it won't be able to run much of anything, as far as AI inference goes).

If you're planning on doing image and video generation, and potentially considering using Strix Halo, then read up on exactly which models run well on ROCm, as opposed to Nvidia CUDA. ROCm is getting better and more supported, but I haven't tested a lot of the other types of AI models aside from LLMs on Strix Halo yet.

Previous Versions
Version 1May 11, 2026 at 00:18

You'd be extremely limited with that machine. Yes, you do need some of that unified memory to run the OS. You'll be stuck running tiny, basically toy models, with very little working space for KV cache. Something like Qwen 3.5 9b will max it out, and don't plan on writing any production code with that model.

Do a Google search, or use ChatGPT to help find machines which will run at least Qwen 3.6 35b and Gemma 4 26b, at least at 4 bit quantization. That's a minimum barrier of entry for getting any actual work completed with self-hosted models. You may still be able to find some sub-$1000 Windows laptops, for example, with a dedicated mobile RTX 3080ti GPU (not the desktop 3080 - the desktop version of that GPU does not have enough VRAM). Those are at the lowest end of usable hardware for any sort of practical LLM inference.

You need a bare minimum of 16 GB VRAM on a dedicated GPU, or at least 32Gb shared RAM on an M series Mac, to do anything useful, but even 32Gb on any of the Macs with unified memory, won't leave you any real room for context, even with heavily compressed models. If you want to stick with Mac, you should really have at least 64Gb unified memory, to get any actual inference work done. You should expect $1350-$1600 for a Mac Studio (M1 Max) with 64Gb or a MacBook Pro 14" (M1 Max).

Consider looking alternately for a tower with an RTX 3090 or dual RTX 3060s. You can find RTX 3060s all over the place for less than $300 each, and 2 of them give you 24Gb VRAM - that's probably still the best buy on the market for running some of the smallest useful models. If you happen to have a motherboard that can support them, that GPU is a no-brainer for price/performance. You're still going to be really constrained with those small GPUs, but the newest Qwen 3.6 and Gemma 4 Mixture of Expert models can run usably fast on them, and they can get tasks done, especially if you're using a lightweight agentic harness like Pi.

A Strix Halo machine like https://www.amazon.com/gp/product/B0DW238TXK will put you into an entirely different class of LLM inference. It's a world of difference. Only look at the 128Gb shared RAM models, if you want the step up in LLM inference performance (do not get a 32GB model - it won't be able to run much).

If you're planning on doing image and video generation, and potentially considering using Strix Halo, then read up on exactly which models run well on ROCm, as opposed to nVidia CUDA. ROCm is getting better and more supported, but I haven't tested a lot of the other types of AI models aside from LLMs on Strix Halo yet.

Back to Thread