
This llama.cpp-based tool streams large mixture-of-experts models directly from NVMe storage when they exceed your RAM or VRAM. It supports DeepSeek-V4.1-Flash-1017 and Qwen3.8 Flash Next with prebuilt binaries for Apple Silicon and Linux CUDA. Benchmarks show significant speedups over standard llama.cpp, reaching 164 tokens per second on an RTX 5090 for 763B parameter models.
The project is young with Windows support still in progress, so it is strictly for macOS or Linux users today. You need fast PCIe 4.0 or 5.0 storage to avoid latency bottlenecks during token generation. If you do not have the specific hardware listed, the performance gains may not justify the setup effort.
more: https://github.com/wici-ai/inferssd
my other side: ivo.blogspot.com
Leave a Reply