Deepseek V4 Flash is quickly becoming my favorite model

The q4-imatrix quant is running on 2 clustered Asus Gx10s at 11.5 tokens per second. This quant is very high quality and is working perfectly in Pi. I wish it was a bit faster, but it's certainly usable at that speed.

This is the closest I've seen my self-hosted models come to near-frontier capability.

I expect to continue using Qwen 3.6 and Gemma 4 for the overwhelming volume of self-hosted inference, because they reliably do very high quality work. The MOE versions of those models perform fast (especially with MTP in Qwen 3.6), and they're capable of completing most basic common coding and agentic tasks.

I'll use Deepseek V4 Flash as a supervisor and helper model, whenever those smaller models get stuck completing a task or otherwise need the deeper knowledge/capability available in V4 Flash (and/or if I don't mind having a task sit and run for longer at ~11.5 tps). I do think it's possible to significantly speed up how this model performs - I'll work on that next.

Below is a reference document which covers the process Pi used to install both the q2 version of V4 Flash on a single GX10 machine, and the q4 version on two clustered GX10s.

Just to note, I do have this setup running successfully over cloudflared, so it's available over HTTPS to any other application on the Internet (just like any OpenAI compatible API endpoint). I covered how to do that at https://aibynick.com/thread/42

────────────────────────────────────────────────────────────────────────────────

DwarfStar (ds4) on ASUS GX10 — Complete Setup Guide

### 1. Single Machine Setup

Clone and build:

bash git clone https://github.com/antirez/ds4.git ~/ds4 cd ~/ds4 make cuda-spark # builds ds4, ds4-server, ds4-agent, ds4-bench, ds4-eval

Download a model:

bash ./download_model.sh q2-imatrix # 81 GB, fits single GX10 (121 GB RAM) ./download_model.sh q4-imatrix # 154 GB, needs 2 GX10s clustered

The ds4flash.gguf symlink points to whichever was downloaded last. Switch manually:

bash ln -sf gguf/DeepSeek-V4-Flash-Q4K* ds4flash.gguf # use q4 ln -sf gguf/DeepSeek-V4-Flash-IQ2XXS* ds4flash.gguf # use q2

Run CLI:

bash ./ds4 --cuda -c 300000

Run server for Pi:

bash ./ds4-server --cuda --ctx 300000 --kv-disk-dir ~/.ds4/kvcache --kv-disk-space-mb 16384

### 2. Pi Integration (Single Machine)

~/.pi/agent/models.json — add the ds4 provider:

json { "providers": { "ds4": { "name": "ds4.c local", "baseUrl": "http://127.0.0.1:8000/v1", "api": "openai-completions", "apiKey": "dsv4-local", "compat": { "supportsStore": false, "supportsDeveloperRole": false, "supportsReasoningEffort": true, "supportsUsageInStreaming": true, "maxTokensField": "max_tokens", "supportsStrictMode": false, "thinkingFormat": "deepseek", "requiresReasoningContentOnAssistantMessages": true }, "models": [ { "id": "deepseek-v4-flash", "name": "DeepSeek V4 Flash (ds4.c local)", "reasoning": true, "thinkingLevelMap": { "off": null, "minimal": "low", "low": "low", "medium": "medium", "high": "high", "xhigh": "xhigh" }, "input": ["text"], "contextWindow": 300000, "maxTokens": 384000, "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 } } ] }, ...existing providers... } }

~/.pi/agent/settings.json — set as default:

json { "defaultProvider": "ds4", "defaultModel": "deepseek-v4-flash", "defaultThinkingLevel": "high", "theme": "dark" }

Auto-start server on boot — user systemd service at ~/.config/systemd/user/ds4-server.service:

```ini
[Unit]
Description=ds4 (DwarfStar) DeepSeek V4 Flash inference server
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/home/gb10a/ds4
ExecStart=/home/gb10a/ds4/ds4-server --ctx 300000 --kv-disk-dir /home/gb10a/.ds4/kvcache --kv-disk-space-mb 16384 --host 127.0.0.1 --port 8000 --cors
Restart=on-failure
RestartSec=10

[Install]
WantedBy=default.target
```

Enable: systemctl --user enable ds4-server.service
Start/stop: systemctl --user start|stop ds4-server.service
Logs: journalctl --user -u ds4-server -f

### 3. Two-Machine Distributed Cluster

Network: Both GX10s connected via fast network port (enp1s0f0np0). Link-local IPs auto-assigned.

Check IPs on each machine:

bash ip -br addr | grep enp1s0f0

Our setup: gx10a = 169.254.10.1, gx10c = 169.254.10.2

Layer split (43 layers total):

┌─────────┬─────────────┬────────────────────┬─────────────┐
│ Machine │ Role │ Layers │ Model slice │
├─────────┼─────────────┼────────────────────┼─────────────┤
│ gx10a │ coordinator │ 0:21 + output head │ ~79 GB │
├─────────┼─────────────┼────────────────────┼─────────────┤
│ gx10c │ worker │ 22:output │ ~74 GB │
└─────────┴─────────────┴────────────────────┴─────────────┘

To run:

```bash
# On both machines first (one-time):
cd ~/ds4 && ln -sf gguf/DeepSeek-V4-Flash-Q4K* ds4flash.gguf

# 1. Start worker on gx10c:
cd ~/ds4 && ./ds4 --cuda --role worker --layers 22:output --coordinator 169.254.10.1 1234 --ctx 200000

# 2. Start CLI coordinator on gx10a (after worker is ready):
cd ~/ds4 && ./ds4 --cuda --role coordinator --layers 0:21 --listen 169.254.10.1 1234 --ctx 200000

# OR start server on gx10a for Pi integration:
cd ~/ds4 && ./ds4-server --cuda --role coordinator --layers 0:21 --listen 169.254.10.1 1234 --ctx 200000 --kv-disk-dir
~/.ds4/kvcache --kv-disk-space-mb 8192
```

Then Pi on gx10a connects to http://127.0.0.1:8000 normally.

### 4. Transferring to Another Machine

Via fast network port (200 Gb/s):

```bash
# Send ds4 source + binaries (skip GGUFs):
rsync -aP --progress --exclude='gguf/' ~/ds4/ user@169.254.x.x:~/ds4/

# Send GGUF model:
rsync -aP --progress ~/ds4/gguf/DeepSeek-V4-Flash-Q4K* user@169.254.x.x:~/ds4/gguf/

# Send Pi configs:
rsync -aP ~/.pi/agent/models.json ~/.pi/agent/settings.json user@169.254.x.x:~/.pi/agent/

# Send systemd service:
rsync -aP ~/.config/systemd/user/ds4-server.service user@169.254.x.x:~/.config/systemd/user/ \
&& ssh user@169.254.x.x "systemctl --user daemon-reload && systemctl --user enable ds4-server.service"
```

Via USB drive (for offline setup): Copy the entire ~/ds4/ directory plus Pi configs and systemd service file. See earlier session
notes for exact file listing.

### 5. Quant Options

┌───────────────┬────────┬─────────────────────────────────────────────────────┐
│ Quant │ Size │ Use Case │
├───────────────┼────────┼─────────────────────────────────────────────────────┤
│ q2-imatrix │ 81 GB │ Single GX10 (121 GB RAM). Leaves 40 GB for KV cache │
├───────────────┼────────┼─────────────────────────────────────────────────────┤
│ q4-imatrix │ 154 GB │ Two GX10s clustered (77 GB each). Best quality │
├───────────────┼────────┼─────────────────────────────────────────────────────┤
│ q2-q4-imatrix │ 98 GB │ Single high-end or clustered mid-range │
└───────────────┴────────┴─────────────────────────────────────────────────────┘

### 6. Performance Reference

┌────────────────────────────┬───────┬──────────────────────────┬────────────┐
│ Setup │ Quant │ Prefill │ Generation │
├────────────────────────────┼───────┼──────────────────────────┼────────────┤
│ Single GX10 (q2) │ 2-bit │ ~31 t/s │ ~11.5 t/s │
├────────────────────────────┼───────┼──────────────────────────┼────────────┤
│ Two GX10s distributed (q4) │ 4-bit │ ~350+ t/s (long prompts) │ ~9-12 t/s │
└────────────────────────────┴───────┴──────────────────────────┴────────────┘

### 7. Files That Need Backing Up

┌───────────────────────────────────────────┬───────────────────────────────────────┐
│ File │ Purpose │
├───────────────────────────────────────────┼───────────────────────────────────────┤
│ ~/ds4/ │ All source, binaries, GGUFs, symlinks │
├───────────────────────────────────────────┼───────────────────────────────────────┤
│ ~/.pi/agent/models.json │ Pi provider config │
├───────────────────────────────────────────┼───────────────────────────────────────┤
│ ~/.pi/agent/settings.json │ Pi default provider/model │
├───────────────────────────────────────────┼───────────────────────────────────────┤
│ ~/.config/systemd/user/ds4-server.service │ Auto-start service │
└───────────────────────────────────────────┴───────────────────────────────────────┘

To restart the distributed cluster from scratch

On both machines first (only needed once — already done):

bash cd ~/ds4 && ln -sf gguf/DeepSeek-V4-Flash-Q4K* ds4flash.gguf

────────────────────────────────────────────────────────────────────────────────

On gx10c (worker) — run this first:

bash cd ~/ds4 && ./ds4 --cuda --role worker --layers 22:output --coordinator 169.254.10.1 1234 --ctx 200000

────────────────────────────────────────────────────────────────────────────────

On gx10a (coordinator) — run this after worker is loaded:

bash cd ~/ds4 && ./ds4-server --cuda --role coordinator --layers 0:21 --listen 169.254.10.1 1234 --ctx 200000 --kv-disk-dir ~/.ds4/kvcache --kv-disk-space-mb 8192

Pi connects to http://127.0.0.1:8000 automatically. If using the CLI instead of the server, replace the last command with:

bash cd ~/ds4 && ./ds4 --cuda --role coordinator --layers 0:21 --listen 169.254.10.1 1234 --ctx 200000