ivo's home

FreeToken is an edge-native inference engine designed to serve frontier-scale Mixture-of-Experts models on personal hardware. It unifies GPU, CPU, and host memory into a single elastic platform to achieve interactive speeds for 290B+ parameter models. A desktop app is available for Windows and Linux to handle the initial setup and tuning.

It is worth a look if you have high-end consumer hardware and need local access to large MoE models without a datacenter cluster. The project is young and relies on a complex hardware abstraction, so expect some configuration friction. It is not a drop-in replacement for production-grade serving stacks like vLLM in a cloud environment.

more: https://github.com/FlashML-org/FreeToken