Here's a prompt generated by Pi, in the session I used to installed Strata on the 5060ti laptop. To install Strata, just paste this prompt into Pi on your machine.
TASK: Install the Strata inference engine and serve Qwen3.8-Flash-Next locally.
Work through the steps in order and tell me what you're doing in plain language.
=== CRITICAL: GET THE RIGHT PROJECT ===
Target: https://github.com/Niko1221/Strata (a C++/CUDA MoE inference engine, MIT)
Do NOT install the PyPI packages named `strata` or `strata-ai`. Despite the name they are
unrelated (a configuration framework and an agent scaffold) and contain no inference code.
The repo's docs/AI_SETUP.md is written specifically for AI agents — read it first and follow it
where it disagrees with this prompt.
=== STEP 1: CHECK THE MACHINE BEFORE DOWNLOADING ANYTHING ===
Requirements (tell me if any is missing, then stop):
GPU : NVIDIA RTX 20/30/40/50 (12 GB+ VRAM), or supported AMD. Needs sm_60+ for NVIDIA.
RAM : 32 GB minimum. RAM decides which model size fits (see Step 3).
Disk : ~80 GB free per model; put it on an SSD.
CPU : x86-64 with AVX2.
Driver: NVIDIA 580+ for the default engine. If older (528-579 is fine), you MUST pass
`--cuda 12` or setup will abort — this uses an opt-in experimental engine.
OS : Windows 10/11 or Linux.
Report GPU/VRAM/driver/RAM/CPU/free disk before continuing. Do NOT install or update GPU
drivers yourself — tell me and I'll do it.
=== STEP 2: CLONE ===
git clone https://github.com/Niko1221/Strata.git
cd Strata
Pick a drive with room. Model files land in a `Strata-data` folder NEXT TO the repo
(override with `--data-dir <path>`).
=== STEP 3: RUN THE BUILT-IN CHECK ===
Windows: python setup.py --check (add --cuda 12 if driver < 580)
Linux : python setup.py --check
The check prints which sizes fit and whether the PC can run Strata. It downloads nothing.
Confirm the "This PC can run Strata." line before continuing.
=== STEP 4: PICK SIZE BY RAM (this is the only real decision) ===
32 GB -> IQ1_M (coder) ... with a 24 GB card, Q2_0 / IQ2_XS also run
48 GB -> IQ2_XS (or Q2_0, the fastest)
64 GB -> IQ2_XS recommended; IQ3_XXS and IQ3_S also fit but only at 64K context
96 GB+ -> IQ3_S, or UD-IQ4_XS (~4-bit)
Approx downloads: Q2_0 66 GB · IQ2_XS 68 GB · IQ3_XXS 76 GB · IQ3_S 84 GB · UD-IQ4_XS 94 GB.
Approx download times: ~2-5 hours at typical home speeds. It is RESUMABLE — do not kill setup
just because it is quiet; rerun the same command to continue.
CONTEXT: setup's default is 32K under 14 GB VRAM, 64K under 20 GB, else 128K. The model's trained
window is 262144 (256K). 384K/512K require EXPERIMENTAL YaRN rope scaling — avoid those unless I
explicitly ask. Pass `--context 262144` only if RAM comfortably allows (KV costs ~13.7 KB/token in
RAM once above 64K, but this frees VRAM for experts).
=== STEP 5: INSTALL — ALWAYS PIN THE SIZE ===
Windows: python setup.py --yes --cuda 12 --family qwen --model <SIZE> --no-start
Linux : ./setup.sh --yes --cuda 12 --family qwen --model <SIZE> --no-start
Set --family/--model from Step 4. --no-start is required so setup doesn't block your shell.
Add `--cuda 12` only if the driver is older than 580.
*** DO NOT RUN `--setup --yes` WITHOUT `--model` ***
By itself, --yes means "take the recommended answer", so setup re-picks the size from scratch and
will start downloading a multi-GB model you did not ask for. Every invocation must name --model.
=== STEP 6: ADD MORE SIZES LATER (optional) ===
Sizes install side by side, each with its own config and start script:
python setup.py --setup --yes --cuda 12 --family qwen --model IQ3_XXS --no-start
python setup.py --setup --yes --cuda 12 --family qwen --model IQ3_S --no-start
Shard 2 (~28.8 GB) is byte-identical between quants — Strata hard-links it, so each extra size
costs roughly (download_gb - 29 GB) of new transfer, not the full amount.
ONLY ONE MODEL CAN RUN AT A TIME: each needs 45-60 GB RAM and all claim port 8080.
=== STEP 7: START ===
Setup prints the script it wrote, named `run-<family>-<size>.bat` / `.sh` (e.g. run-iq2_xs.bat).
Windows: powershell Start-Process -FilePath '.\run-iq2_xs.bat'
Linux : nohup ./run-iq2_xs.sh > strata-server.out 2>&1 &
First start takes 1-3 minutes: it loads 35-55 GB into RAM. The PC may be unresponsive briefly.
This is normal — do not close it.
=== STEP 8: VERIFY (do not skip) ===
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model":"strata","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":64,"reasoning_effort":"none"}'
Health must show "status":"ok" and "loaded":true. Report the model, context, and the
timings.predicted_per_second figure so I can compare speed.
=== STEP 9: HOW TO STOP CLEANLY ===
A supervisor restarts the engine automatically. Killing only the engine process makes it come back.
Kill the whole chain (the cmd/run script, the python server, and the engine exe) together, or run
the matching stop path, then confirm /health fails.
=== STEP 10: TELL ME ===
- Which sizes ended up installed and their context lengths
- Decode speed (tokens/s) per size
- RAM and VRAM in use while running
- The exact start command for each size
- Anything that failed, with the raw error text
SAFETY: keep the default --host 127.0.0.1. Only expose beyond localhost with
`--host 0.0.0.0 --api-key <long-random-secret>`, and give me the key. Never forward a port to a
server with no key. Uses: OpenAI-compatible base URL http://127.0.0.1:8080/v1 (any API key,
any model name); Anthropic at /v1/messages; browser UI at http://127.0.0.1:8080.
```
### Notes for your server
- If it's not Windows, swap START-HERE.bat → ./setup.sh and run-*.bat → run-*.sh. Everything else is identical.
- If that machine has a driver ≥ 580, drop every --cuda 12 and it'll use the supported CUDA 13 engine instead of the experimental one.
- If it has more than 64 GB RAM, you can skip straight to IQ3_S and give it 128K–256K context.
Remember, you'll need at very least 12Gb VRAM and 32Gb RAM to run Strata with the cut down coding model.
64Gb or more RAM will get quants of the full model running fast.