GLM-5.3-Flash on one GPU

Problem

GLM-5.3-Flash is a mixture-of-experts model with 320B parameters, 18B of them active per token: each layer has 288 experts and uses 8 per token. In 4-bit NVFP4 it is 181 GiB; the card has 96 GiB. Plain offloading works, but every token waits on expert weights crossing PCIe.

Approach

• Read experts in wide blocks. Narrow, scattered loads across PCIe were the slow part. A gather kernel, a small GPU program that reads wide and contiguous, took decoding to 13.5 tokens/s.
• Keep the hot experts on the card, not the first ones. In each layer, 35% of the experts get 74% of the picks. So more of the model moved off the card (105 → 125 GiB in host memory) and the freed memory became a cache of the most-used experts: 16.5 tokens/s at peak. The shipped setup caches fewer experts per layer (54 instead of 66) to leave room for long prompts: 15.0 tokens/s.
• Read long prompts in groups. A long prompt touches nearly every expert; staging them 64 at a time took prompt reading from 614 to 1,750 tokens/s.

Results

Decoding went from 10.8 to 15.0 tokens/s as shipped, and 16.5 tokens/s at peak; prompt reading from 614 to 1,750 tokens/s. Most numbers are single runs on one machine, with about ±1 token/s of spread.

What did not work

• Using fewer experts per token reached 20.48 tokens/s, but the output got worse, so it is off.
• Eight bugs stood between a fresh checkout and correct output. One gave no error at all: the model repeated the same token forever.
• One known output defect is not fixed. It affects every NVFP4 mixture-of-experts checkpoint made with the same quantization tool, not only this one.