iFlytek's November second HiCache account concerns a different part of reuse: where cached state lives. The project and operator report extending it from GPU memory into host memory and external storage, taking prefix-cache hits from around thirty percent to ninety-five. Typical inputs ran from thirty thousand to two hundred thousand tokens. Median time to first token was under three seconds afterward, but they don't establish the previous production latency or hardware cost. The hit-rate change is the reported before-and-after comparison.
Moving prefix state beyond GPU memory
An operator account extends cached prefix state into host memory and external storage.
1 source · 1 min read
Link to noteOriginally published 20 November 2026. Reporting, forecasts, prices, odds and event availability reflect the original edition.
Sources & further reading
- Source 1 lmsys.org