← Back to projects Project · 2026 – Present
RWKV SSD-streaming inference engine An RWKV inference engine that streams model weights from SSD when they do not fit in RAM.
A manifest locates each layer in a model pack; a bounded cache and read-ahead path manage weight loading while recurrent state stays in memory. I compared streamed output with a resident run and exposed the engine through a local API.
Engine walkthrough
A token, one layer at a time Watch weights move from storage through the cache and compute path, then into recurrent state.
Your browser does not support embedded video.
Small RAM cache · 3 layers
CPU backend rwkv.cpp native ChatRWKV reference Weight format Grouped U8 Raw dense Read-ahead Next layer · default Adaptive · I/O-bound example Off · demand reads only Sessions 1 session 2 sessions 4 sessions
Token pass 1
Layer Waiting
Weight memory 3 layer cache
Pack reads 0
Cache hits / misses 0 hits · 0 misses
Read-ahead hits 0 hits · 0 misses
Follow one layer through storage, cache, compute, and state.
Playback Guided Slow Fast 01 · SSD storage idle
L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12
Global tensors embedding output head
Selected manifest group blocks.0.*tensor name → codec, shape, offset, length
02 · RAM cache bounded retention
slot 1 empty
slot 2 empty
slot 3 empty
slot 4 empty
slot 5 empty
slot 6 empty
prefetch queue empty
03 · Stage and compute waiting
A Decode / prepare prepare current layer for the backend
B Install active layer set current weights in the execution path
C Run RWKV block time mix, channel mix, and layer state
01 Time mix previous activation + recurrent matrix
↓ 02 Channel mix feed-forward work + previous activation
↓ 03 Save state save the updated state for each session
L1 TMix CMix
L2 TMix CMix
L3 TMix CMix
L4 TMix CMix
L5 TMix CMix
L6 TMix CMix
L7 TMix CMix
L8 TMix CMix
L9 TMix CMix
L10 TMix CMix
L11 TMix CMix
L12 TMix CMix
token activation → L1 → next layer
04 · Recurrent state resident
Session state ready each layer keeps its own recurrent state session 1 · state ready
output head · logits waiting
Execution trace 0 / 7
01 Find layer record manifest supplies tensor names and file offsets
02 Read requested weights fetch a pack span, or reuse resident, cached, or queued data
03 Check cache and prepare reuse, decode, or keep supported weights packed
04 Install active weights native path runs a layer-local graph
05 Time mix normalize activation, read prior state, run the recurrent attention step
06 Channel mix add the residual, normalize, then run the feed-forward step
07 Save state and release advance all sessions; drop transient weights or retain them within the cap
Choose a mode or setting, then run the next stage to begin.
Recent events simulated
Engine ready. Choose a mode, then start the pass. Read-ahead can overlap block work stage order · no time scale
Stage Find Read Prepare Install TMix CMix Save Block Read-ahead
Read-ahead begins during the current block. This example makes the queued spans ready before the next block asks for them.
Global weights embedding + output head prepared before this token pass
Retained block weights 0 / 3 cache slots
In-flight work no active layer active layer and read-ahead are additional
Session state 1 × 12 layer states separate state, shared weight sweep
Illustrative 12-layer pass. Read-ahead and counters are simulated. Backend docs · Streaming docs · Read-ahead code .
Measurement
Weight loading One controlled comparison · 0.1B model
Warm decode throughput Native CPU, 0.1B model, 64 output tokens, five samples per profile. This compares residency strategies within one local setup; it does not establish performance across hardware or model sizes.
Measurement
Multiple requests Same model pack · concurrent local sessions
Combined completion speed Warm CPU test, 16 generated tokens per request. Ratios compare combined completion with running the same requests sequentially; paired output text matched.