Writing
Most of these started as an internal document I did not want to write, about a number I did not like. The numbers are measured unless the text says otherwise.
Notes
The billion-token question
Sep 2026
Free tiers cap requests, not tokens. That single fact turns the
question into an arithmetic problem about how to build a 35,000-token
request 28,800 times a day — and why padding is not an answer.
What a 304B model actually costs to self-host
Sep 2026
Two B300s serving a sparse MoE: 6,972 tok/s aggregate at 100
concurrent streams, a million-token context in 72.8 seconds, the
DeepGEMM crash that ate a week, and the request-layer bug that made
the model look stupid.
Prompt cache is the biggest lever on your agent bill
Aug 2026
50.6B prompt tokens in 30 days, $1,799 paid, $10,629 saved. Why a
blended 88.5% hit rate is the wrong number, the four things that
break a prefix, and the half-second of latency that came free.
When your agent lies about success
Sep 2026
A close event during timeout cleanup won a race, and the shell tool
reported exit code zero on a command that was killed. Why a false
success is worse than a crash, and how to close the whole class.
Fine-tuning a 304B MoE on a $500 ceiling
Aug 2026
Every gate passed except the one that mattered. Roughly 263 GiB
allocated per GPU during BF16 expert aggregation, and what it means
that a model loaded for inference and could not load to train.
A 149M router for 455 skills
Sep 2026
An audit over 346 real sessions found the remote router missing 37%
of tool-bearing turns and 40% of them mid-turn. Here is the
right-sized local encoder that replaced it, and why it runs offline
in under 30 ms.