What a 304B model actually costs to self-host

The hardware decision, briefly

Running your own endpoint only makes sense when you have a workload that is either privacy-bound, latency-bound, or cheap enough that per-token pricing is embarrassing. Ours was the first two. What I did not expect was how much of the value would come from being able to control the request layer.

We compared renting GPU time. Vast was about 5% cheaper on comparable hardware, but it came with nine-to-twenty-eight-day leases, host variability, bandwidth charges and forced migrations. RunPod's secure cloud gives persistent volumes, which matters enormously when your model is 156 GiB on disk and your container filesystem is wiped every time the pod stops. Their serverless product was out immediately: one GPU per worker, and this model needs about 200 GB resident.

Observed pricing at the time, per GPU hour:

Two B300s is $15.78 an hour plus disks, call it $15.86. That is the real number to hold in your head while reading the rest of this, because it turns every benchmark into a cost.

There is no full-precision checkpoint

The first thing that breaks your assumptions is precision. The model we ran — DeepSeek V4 Flash, the 0731 checkpoint — is released natively as MXFP4 routed-expert weights with FP8 everywhere else: attention, router, norms, and an FP8 KV cache for the MLA path. Roughly 304B total parameters, 13B active, 256 routed experts plus one shared expert, six chosen per token, 43 layers, about 156 GiB across 48 safetensor shards.

There is no official BF16 or FP16 checkpoint. The torch_dtype: bfloat16 line in the config describes compute dtype, not stored weights. Dequantising the whole thing to BF16 inflates it past what two cards will hold. Everything downstream — fine-tuning frameworks, quantization toolkits, your mental model of "load the model in half precision" — inherits a wrong assumption from that one line.

The DeepGEMM saga

The single biggest debugging cost of the project was a startup crash that arrived late in engine initialisation, after a five-minute model load, wearing this message:

Corrupted JIT cache

Two theories came and went. First suspect: the XFS volume on the persistent disk. Disproven. Second suspect: the eight-thread safetensor prefetch racing during load. Disproven. The actual answer was the runtime version. vLLM 0.23 has a deterministic startup crash on B300 in the DeepGEMM path; moving to 0.25.0 made it disappear and never come back.

The stack that worked, for anyone starting from zero:

• vLLM 0.25.0
• Torch 2.11.0 + cu130, CUDA 13
• FlashInfer 0.6.13
• vLLM Router 0.1.15, cache-aware policy

Install it inside the persistent volume, not the container. A stopped pod erases the container's Python environment, and reinstalling a CUDA-linked stack every restart is a tax you pay forever.

Verify the interconnect before you design anything

On a two-GPU node, whether the cards are actually linked changes which parallelism strategies are even valid. Check it, do not assume it:

nvidia-smi topo -m        # GPU0<->GPU1 must show NV18
nvidia-smi topo -p2p n
nvidia-smi nvlink --status

Anything reporting PIX, PXB, PHB, NODE or SYS is a PCIe path, and any profile that depends on NVLink should fail closed rather than silently degrade. Ours reported NV18: eighteen links at 53.125 GB/s, about 956 GB/s per direction.

What the endpoint actually does

All of these ran through the public TLS gateway, into the cache-aware router, into two replicas with 64 sequence slots each. Prompt caching on, thinking disabled unless stated.

Those rows are a burst test: one request per concurrent client, 128 maximum output tokens. Pushing to 100 concurrent clients with a 400-token cap moved the aggregate to 6,972 tok/s, with 116 tok/s per request, p50 time-to-first-token of 1.86s and p95 total latency of 5.59s. All 100 requests completed.

This is the row that kills a common misconception. When a provider advertises "about 3,000 tokens per second," they mean system aggregate across all users, not the speed of your request. Our two-card system passed 3,000 aggregate in both 100-user tests while each individual user was getting somewhere near 100 tok/s.

Reasoning mode changes the shape of the load

With thinking enabled, the same 100-concurrent test produced 6,520 aggregate tok/s and 104.6 tok/s per user, with p50 time-to-first-token at 1.51s. Aggregate looks similar. Latency does not: median total latency went from 5.3s to 12.2s, because the model is generating a long internal monologue before it answers. Reasoning is not a quality switch, it is a throughput reallocation.

Long context is a different product

We calibrated prompt sizes through the real tokenizer rather than trusting a character count. Two results:

• a 131,072-token prompt was accepted and completed in 6.06s;
• a 1,048,000-token prompt — 576 tokens below the model's 1,048,576 limit, leaving headroom for output and protocol — completed in 72.86s.

Both returned CONTEXT_OK. The second number is the honest one to quote for "a million tokens": it is real, and it is not interactive. If your product requires a million-token prompt, your product requires a user who will wait over a minute.

The best bug was not in the GPU

We got a report that the model "feels dumber than other providers." It was not. The weights were identical. There were two request-layer defects, and the first one is my favourite bug of the year.

Clients were sending OpenRouter-style reasoning.effort: medium. The model exposes low | high | max. Somewhere in translation, medium was mapped down to low. Every user who asked for medium effort silently received the cheapest reasoning tier available, and then concluded the model was weak.

The fix was an explicit mapping, written down:

low        -> low
medium     -> high
high       -> high
xhigh/max  -> max
enabled:false -> thinking:false

The second defect was temperature: 0. Our generic client sent temperature zero for custom endpoints. For a reasoning model that is not "deterministic," it is a different distribution, and it shows up as flatter, more repetitive output.

Wiring it into a desktop client

Three traps, all of which produced the same unhelpful error message:

• CORS is mandatory. The desktop app's provider test runs in a renderer process doing a browser-style fetch. curl succeeds while the app reports the endpoint is invalid, until the gateway sends the right headers.
• Model ID must be exact. The served name was deepseek-v4-flash-0731 with no vendor prefix. Selecting a prefixed hosted ID silently routed to a different provider.
• Port matters. The gateway lives on its own port, not the one your notebook server uses. Choosing the wrong one gives you a valid-looking endpoint that answers nothing.

So what does it cost?

At $15.86 an hour for the pair, and roughly 7,000 aggregate tokens per second sustained, you are paying somewhere near $0.63 per million tokens at full utilisation — and considerably worse at idle, which is most of the time. That is the whole economics of self-hosting in one sentence: the marginal token is cheap, the floor is expensive, and the floor is what kills you unless you are actually saturating the box.

What you buy with that floor is control. I can pick the reasoning tier mapping. I can decide the retry policy. I can watch the request log. I can pin an exact revision and know it will answer the same way in six months. For a platform that has to explain its own behaviour, that is worth more than the per-token saving.