0

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

https://huggingface.co/blog/moe-expert-streaming(huggingface.co)
Running massive Mixture-of-Experts (MoE) models with over 180 billion parameters on consumer hardware like a 32GB MacBook is a major challenge due to memory limitations. A new technique developed for the Transformers library tackles this by streaming only the necessary model "experts" directly from an SSD as they are needed for each token. This method cleverly uses a preallocated "slot pool" in memory, which allows existing, highly optimized computation kernels to run without modification. The approach dramatically improves efficiency, achieving token generation speeds close to specialized engines and making huge models practical on local machines.
0 points•by ogg•5 hours ago

Comments (0)

No comments yet. Be the first to comment!

Have an account? Log in to join the discussion.