Strata w/ Qwen 3.8 Flash Next has shockingly great performance on inexpensive GPUs with 12Gb+ VRAM. It's the BEST price/performance solution yet for local self-hosting.

Nick AntonaccioAdmin
Oct 11, 2026 at 22:46 (edited, 3 revisions)

If you have at least 12Gb of VRAM on your local GPU, and at least 32Gb system RAM, point your agent to this repository:

That's Strata, an inference engine which implements a version of the Qwen 3.8 Flash Next model on small GPUs. That's a 125 billion parameter LLM, which is much smarter, more capable, and much faster than all those versions of Qwen 3.8 27b you've been trying to make work:

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

Here's a good video demonstrating it:

https://www.youtube.com/watch?v=B90V3IURGpE

History

Nick AntonaccioAdmin
Oct 11, 2026 at 22:53 (edited, 14 revisions)

Strata isn't playing around. I think it's the most significant recent engine advancement available to the self-hosting community. It enables an entirely new threshold of capability on underpowered machines. To run it well, you do need more RAM than is required by other engines, but RAM is far cheaper than VRAM and GPU hardware, and the price/performance improvement made possible by this engine + the QFN 3.8 architecture, is spectacular.

I'm getting the following performance results on an old laptop with a mobile RTX 3080ti:

Qwen 3.8 Flash Next IQ2_XS runs at 70+ tokens per second with 256k KV cache (holy crap, holy crap that's fast - and don't knock the q2, it's very capable)
IQ3_XS runs at ~48 tokens per second with 64k context
IQ3_S seems to average around ~40 tokens per second with 64k context

That's astounding. Here's a screen shot of the monitoring page during a long context task:

https://com-pute.com/nick/strata_q38fn_60tps_on_3080ti_mobile.png

And here's a quick demo of a 3D car racing demo game created with the IQ2_XS quant (Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS):

https://com-pute.com/nick/car-racing-game--qwen38fn-iq2xxs-strata-on-mobile-3080ti.html

So, yep, even that quant is capable, and while building the app above, I regularly saw the generation speed top 70 tokens per second! It seemed to average around 60-ish tokens per second, even with 1/2 of the significant 256k KV cache filled! 😲

Here are Rubik's Cube examples built by both the q2 and q3 quants:

Critically, the test machine I used above had 64Gb RAM installed, which is what makes Strata really zoom. You can run a cut-down coding version of Qwen 3.8 Flash, if you have only 32Gb, but for the full Strata experience, you'll need at least 64Gb RAM (not VRAM, regular system RAM - that's the difference with Strata, it can get by with very little VRAM, if you've got enough RAM).

The numbers above mean that if you have a GPU with 12Gb+ VRAM, Strata should likely replace your Qwen 3.8 27b implementations.

BTW, people often don't realize that the laptop 'ti' version of the 3080 GPU, which I used to test and build the examples above, has 16Gb of VRAM (as opposed to 10 or 12, like the desktop version), so I think it's more likely to get overlooked in the used hardware market. As far as I know, it's the only exception in which an Nvidia laptop version of a GPU had more VRAM than the same-named desktop version.

Previously, the only really useful LLM which ran fast enough on that old test machine above was Qwen 3.6 35a3 (MOE), and it only reached ~14 tokens per second. That's one hell of a dramatic improvement to both speed and capability.

To give you another comparison, the well known GSQ-RCO optimized version of Qwen 3.8 27b did run on this hardware, but at a nearly unusable slow speed of only 5.56 tokens per second. Those popular 27b variations are much less capable, less knowledgeable, and now much slower compared to Strata running Qwen 3.8 Flash Next.

To go from numbers like that, with the 27b and 35a3 models, to 70 tokens per second, using a far more capable 125 billion parameter model, is a mind-blowing universe of difference.

The Qwen 3.8 Flash Next model gets really close to delivering the sort of LLM capabilities which previously required DGX Spark, Strix Halo, and other machines with 96-128Gb VRAM (and I typically use 2 or more of those machines clustered together, for comparable inference quality and speed). The QFN 3.8 model is strikingly capable, even at IQ2_XXS - and at IQ3_S, it achieves the same benchmark scores as the full 16 bit version.

The whole situation is honestly shocking, and it changes the hardware game significantly. This enables machines that can currently be put together for well under $1000, to work as powerful AI inference systems.

Be aware that Strata is currently tied to the Qwen 3.8 Flash Next model, but 3rd-party forks have been released which run Qwen 3.6 35a3 - I'll test those next. I have to wonder if the industry may be able to get models like 3.6 35a3 running on cheap phones, as the Strata approach matures...

As it currently exists, Strata makes machines with relatively inexpensive GPUs, genuinely capable modern inference monsters.

This is the sort of self-hosted capability everyone has hoped for. Buy a single RTX 5060 or even a 3060, upgrade to at least 64Gb system RAM, and you can build a very capable system in the $700-$1000 range.

Spend an extra $120-$200 for a total of 128Gb RAM (costs vary depending on whether your motherboard uses DDR4 or DDR5), and you can run far less quantized versions of Qwen 3.8 Flash Next, even faster in Strata, than you can with 64Gb.

Or if one of those mobile RTX 3080t laptops pops up on Ebay, anywhere in the $1000 range, they can be an incredible buy, because it's not so well known that they have 16Gb VRAM.

Either way, if you're looking for AI hardware, RAM costs only a small fraction of the VRAM in expensive GPUs, even at today's prices.

Currently Strata is best employed for single user inference (multiple simultaneous users will slow it down), but nothing else comes close to its price/performance ratio, for bottom of the barrel AI hardware. What an absolutely spectacular solution for self-hosting a powerful LLM.

History

Nick AntonaccioAdmin
Oct 11, 2026 at 19:11 (edited, 4 revisions)

Here's a prompt generated by Pi, in the session I used to installed Strata on the 5060ti laptop. To install Strata, just paste this prompt into Pi on your machine.

   TASK: Install the Strata inference engine and serve Qwen3.8-Flash-Next locally.
   Work through the steps in order and tell me what you're doing in plain language.

   === CRITICAL: GET THE RIGHT PROJECT ===
   Target: https://github.com/Niko1221/Strata  (a C++/CUDA MoE inference engine, MIT)
   Do NOT install the PyPI packages named `strata` or `strata-ai`. Despite the name they are
   unrelated (a configuration framework and an agent scaffold) and contain no inference code.
   The repo's docs/AI_SETUP.md is written specifically for AI agents — read it first and follow it
   where it disagrees with this prompt.

   === STEP 1: CHECK THE MACHINE BEFORE DOWNLOADING ANYTHING ===
   Requirements (tell me if any is missing, then stop):
     GPU   : NVIDIA RTX 20/30/40/50 (12 GB+ VRAM), or supported AMD. Needs sm_60+ for NVIDIA.
     RAM   : 32 GB minimum. RAM decides which model size fits (see Step 3).
     Disk  : ~80 GB free per model; put it on an SSD.
     CPU   : x86-64 with AVX2.
     Driver: NVIDIA 580+ for the default engine. If older (528-579 is fine), you MUST pass
             `--cuda 12` or setup will abort — this uses an opt-in experimental engine.
     OS    : Windows 10/11 or Linux.
   Report GPU/VRAM/driver/RAM/CPU/free disk before continuing. Do NOT install or update GPU
   drivers yourself — tell me and I'll do it.

   === STEP 2: CLONE ===
     git clone https://github.com/Niko1221/Strata.git
     cd Strata
   Pick a drive with room. Model files land in a `Strata-data` folder NEXT TO the repo
   (override with `--data-dir <path>`).

   === STEP 3: RUN THE BUILT-IN CHECK ===
     Windows: python setup.py --check            (add --cuda 12 if driver < 580)
     Linux  : python setup.py --check
   The check prints which sizes fit and whether the PC can run Strata. It downloads nothing.
   Confirm the "This PC can run Strata." line before continuing.

   === STEP 4: PICK SIZE BY RAM (this is the only real decision) ===
     32 GB  -> IQ1_M (coder) ... with a 24 GB card, Q2_0 / IQ2_XS also run
     48 GB  -> IQ2_XS  (or Q2_0, the fastest)
     64 GB  -> IQ2_XS recommended; IQ3_XXS and IQ3_S also fit but only at 64K context
     96 GB+ -> IQ3_S, or UD-IQ4_XS (~4-bit)
   Approx downloads: Q2_0 66 GB · IQ2_XS 68 GB · IQ3_XXS 76 GB · IQ3_S 84 GB · UD-IQ4_XS 94 GB.
   Approx download times: ~2-5 hours at typical home speeds. It is RESUMABLE — do not kill setup
   just because it is quiet; rerun the same command to continue.

   CONTEXT: setup's default is 32K under 14 GB VRAM, 64K under 20 GB, else 128K. The model's trained
   window is 262144 (256K). 384K/512K require EXPERIMENTAL YaRN rope scaling — avoid those unless I
   explicitly ask. Pass `--context 262144` only if RAM comfortably allows (KV costs ~13.7 KB/token in
   RAM once above 64K, but this frees VRAM for experts).

   === STEP 5: INSTALL — ALWAYS PIN THE SIZE ===
     Windows: python setup.py --yes --cuda 12 --family qwen --model <SIZE> --no-start
     Linux  : ./setup.sh   --yes --cuda 12 --family qwen --model <SIZE> --no-start
   Set --family/--model from Step 4. --no-start is required so setup doesn't block your shell.
   Add `--cuda 12` only if the driver is older than 580.

   *** DO NOT RUN `--setup --yes` WITHOUT `--model` ***
   By itself, --yes means "take the recommended answer", so setup re-picks the size from scratch and
   will start downloading a multi-GB model you did not ask for. Every invocation must name --model.

   === STEP 6: ADD MORE SIZES LATER (optional) ===
   Sizes install side by side, each with its own config and start script:
     python setup.py --setup --yes --cuda 12 --family qwen --model IQ3_XXS --no-start
     python setup.py --setup --yes --cuda 12 --family qwen --model IQ3_S   --no-start
   Shard 2 (~28.8 GB) is byte-identical between quants — Strata hard-links it, so each extra size
   costs roughly (download_gb - 29 GB) of new transfer, not the full amount.
   ONLY ONE MODEL CAN RUN AT A TIME: each needs 45-60 GB RAM and all claim port 8080.

   === STEP 7: START ===
   Setup prints the script it wrote, named `run-<family>-<size>.bat` / `.sh` (e.g. run-iq2_xs.bat).
     Windows: powershell Start-Process -FilePath '.\run-iq2_xs.bat'
     Linux  : nohup ./run-iq2_xs.sh > strata-server.out 2>&1 &
   First start takes 1-3 minutes: it loads 35-55 GB into RAM. The PC may be unresponsive briefly.
   This is normal — do not close it.

   === STEP 8: VERIFY (do not skip) ===
     curl http://127.0.0.1:8080/health
     curl http://127.0.0.1:8080/v1/models
     curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
       -d '{"model":"strata","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":64,"reasoning_effort":"none"}'
   Health must show "status":"ok" and "loaded":true. Report the model, context, and the
   timings.predicted_per_second figure so I can compare speed.

   === STEP 9: HOW TO STOP CLEANLY ===
   A supervisor restarts the engine automatically. Killing only the engine process makes it come back.
   Kill the whole chain (the cmd/run script, the python server, and the engine exe) together, or run
   the matching stop path, then confirm /health fails.

   === STEP 10: TELL ME ===
   - Which sizes ended up installed and their context lengths
   - Decode speed (tokens/s) per size
   - RAM and VRAM in use while running
   - The exact start command for each size
   - Anything that failed, with the raw error text

   SAFETY: keep the default --host 127.0.0.1. Only expose beyond localhost with
   `--host 0.0.0.0 --api-key <long-random-secret>`, and give me the key. Never forward a port to a
   server with no key. Uses: OpenAI-compatible base URL http://127.0.0.1:8080/v1 (any API key,
   any model name); Anthropic at /v1/messages; browser UI at http://127.0.0.1:8080.
 ```

 ### Notes for your server

 - If it's not Windows, swap START-HERE.bat → ./setup.sh and run-*.bat → run-*.sh. Everything else is identical.
 - If that machine has a driver ≥ 580, drop every --cuda 12 and it'll use the supported CUDA 13 engine instead of the experimental one.
 - If it has more than 64 GB RAM, you can skip straight to IQ3_S and give it 128K–256K context.

Remember, you'll need at very least 12Gb VRAM and 32Gb RAM to run Strata with the cut down coding model.

64Gb or more RAM will get quants of the full model running fast.

History

Nick AntonaccioAdmin
Oct 11, 2026 at 17:37

This post has been deleted.

Nick AntonaccioAdmin
Oct 11, 2026 at 17:38

This post has been deleted.

Nick AntonaccioAdmin
Oct 11, 2026 at 19:02 (edited, 1 revision)

This post has been deleted.

History

Nick AntonaccioAdmin
Oct 11, 2026 at 19:00

Below is a prompt to set up Pi on a remote machine, to use models served by Strata on your locally hosted server as the LLM provider. I run Cloudflared on my servers to tunnel connections to my servers, so having that set up for your server machine(s) is a prerequisite (unless you want to forward ports, or simply use your servers inside a closed local area network). See https://aibynick.com/thread/42 to understand how I set up Cloudflared tunnels.


   Connect this machine to a remote Strata inference server (Qwen3.8-Flash-Next, 125B MoE) exposed
   through a Cloudflare tunnel.

     BASE_URL = <https://yourroute.yourdomain.com>
     API_KEY  = <yourapikey>

   It is an OpenAI-compatible API; app base URL is BASE_URL + "/v1".
   An Anthropic-compatible endpoint is at BASE_URL + "/v1/messages".
   The browser UI is at BASE_URL (enter the key under About > Settings).

   === MODEL NAME: SET IT ONCE, NEVER CHANGE IT ===
   The server IGNORES the model name entirely. It always serves whatever model the host machine
   currently has loaded. Send a fixed constant such as "strata" for every request and leave it
   permanently. Do NOT try to select a model by name, and do NOT reconfigure this client when the
   host changes models — nothing on this side needs to change.

   === SWITCHING MODELS HAPPENS ON THE HOST, NOT HERE ===
   The host operator switches between three sizes (IQ2_XS, IQ3_XXS, IQ3_S). They all share the same
   URL, port and key. Your configuration is stable across every switch.

   The ONLY thing that changes for you is the usable context window, because the host's available
   context differs per model:
       IQ2_XS   -> 262,144 tokens
       IQ3_XXS  ->  65,536 tokens
       IQ3_S    ->  65,536 tokens
   If you send more than the current ceiling the server REFUSES the request (it never silently
   truncates). Handle this gracefully: on such an error, trim or summarise your context and retry,
   and tell me the model appears to have changed. Query BASE_URL/v1/models to see which model is
   currently loaded — treat its "id" as informational only, never as something to route on.

   Also expect 30-90 seconds where requests FAIL whenever the host switches models.

   === SETUP AND VERIFICATION ===
   1. Confirm auth is enforced:
        curl -s -o /dev/null -w "%{http_code}\n" BASE_URL/v1/models                            # expect 401
        curl -s -H "Authorization: Bearer $API_KEY" BASE_URL/v1/models                          # expect 200
   2. Confirm inference works; report timings.predicted_per_second and the reported model id:
        curl -s BASE_URL/v1/chat/completions -H "Authorization: Bearer $API_KEY" \
          -H "Content-Type: application/json" \
          -d '{"model":"strata","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":64,"reasoning_effort":"none"}'
   3. Ask me which client to configure (OpenAI SDK, Continue, Cline, aider, Claude Code, plain
      script) before installing anything, then set base URL BASE_URL + "/v1" and the API key.
   4. Use reasoning_effort "none" or "low" unless I need deep reasoning — "high" is the default
      and is substantially slower.

   === THINGS TO EXPECT AND TELL ME ABOUT ===
   - The server handles ONE request at a time; concurrent agents serialise. Don't fan out in parallel.
   - Prefill is slow for large first prompts (~1 minute per 30K tokens). Short chat is fine; dumping
     a large codebase as context will feel slow.
   - A home GPU behind a tunnel: expect variable latency (measured 27-34 tokens/s decode).
   - 403 "Host is not allowed" means the tunnel hostname is untrusted — report it, don't work around it.
   - Do not send secrets you would not share with the host machine.

Nick AntonaccioAdmin
Oct 11, 2026 at 19:23

And with that post, this video, about the risks of installing blindly via prompts in an agent, is worth watching:

https://www.youtube.com/watch?v=i18vpg0T-mk

His recommended download, to install it is:

https://github.com/robertkarl/UnbiasedStrata

Nick AntonaccioAdmin
Oct 12, 2026 at 13:56 (edited, 4 revisions)

I've got Strata running the IQ3_S quant of Qwen 3.8 Flash Next on Strix Halo machines, with 256k KV cache (context length), at a maximum of over 40 tokens per second for short prompts, in Windows 11. It seems to average around ~35 tokens per second on very long prompts.

That IQ3_S version of the model is what Strata has documented as achieving the same benchmark scores as the full precision version.

Strata is not optimized for Strix Halo, so this is a very happy result - faster than all the other versions of the QFN model I've run in LM Studio and other engines in Windows, and the quality is definitely very good.

Clearly, Strata and the QFN 3.8 model are built to take advantage of the CUDA stack. Even the old RTX 3080ti mobile GPU, with only 16Gb VRAM and 64Gb RAM performs faster than the Strix Halo. It's not really apples to apples, though, because the Strix Halo above was configured to enable 4x the context length at IQ3_S (highest quality). That's where all that Strix Halo VRAM comes in handy.

I love that users don't need to wipe Windows and install Linux on Strix Halo to get very good performance from Strata - though the fastest solutions on Strix Halo are still possible only with Linux installed.

Still, the IQ2_XS quant that runs 256k context on the mobile 3080ti in Windows with 64Gb RAM is quite capable & knowledgeable, and you can buy the hardware to run it for about 1/3 the cost of any Stix Halo machine - it's actually affordable, even for casual hobbyists (~$1000 range, where the next step up to clustered DGX Spark machines is now $14,000-$20,000). Add a few hundred dollars of RAM, and you can run IQ3_S at 256k on machines with inexpensive RTX GPUs. So there are options and alternatives to support your budget and preferences.

I'm becoming a huge fan of Strata :)

History

Nick AntonaccioAdmin
Oct 12, 2026 at 02:35

#10

Here's an installation prompt for Strix Halo:


TASK: Install and serve the Strata inference engine with Qwen3.8-Flash-Next on this machine.

CRITICAL - GET THE RIGHT PROJECT
Target: https://github.com/Niko1221/Strata  (C++/CUDA/HIP MoE inference engine, MIT).
Do NOT install the unrelated PyPI packages `strata` or `strata-ai` (a config framework and
an agent scaffold, no inference code). Read the repo's docs/AI_SETUP.md first and follow it
where it disagrees with this prompt. docs/STRIX_HALO.md is the hardware-specific doc.

EXPECTED HARDWARE (verify before downloading anything)
- ASUS ROG Flow Z13 GZ302EA, AMD Ryzen AI MAX+ 395, Radeon 8060S iGPU (gfx1151, Strix Halo),
  Windows 11.
- 128 GB LPDDR5X (8 x 16 GB). Windows reports only ~32 GB visible because ~96 GB is reserved
  by the BIOS as the iGPU UMA carve-out. This is expected, not a fault. Confirm installed RAM:
    powershell -NoProfile -Command "Get-CimInstance Win32_PhysicalMemory | Measure-Object Capacity -Sum"
  Expect ~128 GB. The GPU reports ~107.9 GiB usable.
- No NVIDIA GPU. Do NOT pass --cuda. Do NOT install or change any driver.
- ~80+ GB free on an SSD. CPU x86-64 with AVX2/AVX-512.

STEP 1 - Get the source
GitHub can be very slow from some networks (git clone may time out). Prefer the official tarball:
  curl -sL -o strata.tar.gz https://codeload.github.com/Niko1221/Strata/tar.gz/refs/heads/main
  tar xzf strata.tar.gz
  mv Strata-main Strata
(If git clone works fine, that is acceptable too.)

STEP 2 - Check the PC (this downloads no model)
  cd Strata
  python setup.py --check
Confirm the line: "This PC can run Strata."
It should detect: AMD Radeon 8060S, unified memory, gfx1151, "32 GB seen by the OS + a 96.0 GB
BIOS carve-out", and list which sizes fit.

STEP 3 - Install, pinning the size explicitly
  START-HERE.bat --yes --family qwen --model IQ3_S --context 262144 --no-start
- Do NOT add --cuda (this is an AMD machine).
- Do NOT run --yes without --model; --yes alone re-picks the size and can start a large
  download you did not ask for.
- 262144 is the model's full trained window (no YaRN needed).
- The model comes from Hugging Face (repo ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF),
  ~84 GB total, resumable. A 2-4 hour download is normal; do not kill it because it is quiet.
- Model files land in a `Strata-data` folder next to the repo.
- On Windows AMD, setup uses the ready-made HIP engine built for gfx1151 and checks the
  device before starting the model download.

STEP 4 - Start
  powershell -Command "Start-Process -FilePath '.\run-iq3_s.bat' -WorkingDirectory (Get-Location)"
First start takes 1-3 minutes (it loads ~50 GB of experts into the GPU carve-out). The machine
may be briefly unresponsive; do not close it.

STEP 5 - Verify
  curl http://127.0.0.1:8080/health
  curl http://127.0.0.1:8080/v1/models
  curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"strata\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hello in five words.\"}],\"max_tokens\":64,\"reasoning_effort\":\"none\"}"
Health must show "status":"ok" and "loaded":true. Model id: qwen3.8-flash-next-iq3_s,
context 262144. Expected decode: roughly 10-45 tokens/s depending on prompt length and MTP.

STEP 6 - Secure before exposing (REQUIRED)
Keep the server bound to 127.0.0.1. If this machine is reachable through a tunnel or proxy,
you MUST set an API key and whitelist the tunnel host:
  1. Generate a strong key:
       python -c "import secrets;print('sk-strata-'+secrets.token_urlsafe(36))"
  2. Add to strata-iq3_s.json:   "api_key": "<KEY>"
     and:                        "allowed_hosts": ["<TUNNEL_HOST>"]
     (replace <TUNNEL_HOST> with this machine's Cloudflare/tunnel hostname)
  3. Restart the chain and verify: no key -> 401, correct key -> 200.
Never expose a server with no key. If asked to listen beyond localhost, use
--host 0.0.0.0 TOGETHER WITH --api-key <secret>, and give the user the key.

STEP 7 - Stop cleanly
The server is a process chain: cmd (run-iq3_s.bat) -> python serve/server.py -> python -> strata.exe.
A supervisor can restart the engine, so kill the whole tree, not just strata.exe:
  taskkill /PID <run-iq3_s.bat cmd PID> /T /F
Then confirm /health fails (connection refused).

REPORT BACK TO THE USER
- Specs verified (GPU/VRAM/driver/RAM/CPU/free disk).
- Which sizes ended up installed and their context lengths.
- Decode speed (tokens/s).
- RAM and VRAM in use while running.
- The exact start command.
- Anything that failed, with the raw error text.

Please login to post a reply.