llama.cpp PR Caches ‘Hot’ MoE Experts for Performance Boost

The latest llama.cpp PR introduces a caching mechanism for MoE experts, significantly boosting token generation speeds for developers using consumer GPUs. It shows promise in enhancing performance but comes with limitations. This feature is particularly relevant for indie developers and small teams working with AI models.

What happened

The original report discusses a new pull request (PR #26563) for llama.cpp that implements a heatmap to track frequently used Mixture of Experts (MoE) models. This PR aims to optimize GPU usage by caching the most active experts in VRAM while less frequently used experts remain on the CPU. Early tests show notable performance improvements in token generation speeds.

According to the report, when testing with the Qwen3.6-35B-A3B model and 8GB of VRAM, the performance jumped from 33.25 tokens per second to 56.0 tokens per second. Another configuration saw an increase from 17.34 to 35.93 tokens per second. However, not all configurations benefited; some were slower with caching enabled, indicating that this solution may not be universally applicable.

Why it matters for builders

This new feature in llama.cpp is significant for builders who are working with large AI models on consumer-grade GPUs. It enables better resource management and can lead to more efficient application performance. Developers looking to enhance their AI model’s speed without needing high-end hardware will find this particularly useful.

The details

• Performance Boost: The caching mechanism reportedly increases token generation speeds by up to 2.07x, depending on the model.
• GPU Optimization: Frequently used MoE experts are stored in VRAM, reducing the load on CPU for less active experts.
• CUDA Compatibility: Currently, this feature only supports CUDA, limiting its use to compatible hardware.
• Single-Token Decoding: The caching feature is active only during single-token decoding, which may restrict its application in multi-token contexts.
• Ongoing Development: The PR is still open and has not yet been merged, meaning it is subject to further changes and testing.

Compared to other optimization methods, this caching approach targets specific usage patterns, which may yield better results under certain conditions.

The catch

This new caching feature is not without limitations. It currently only operates with CUDA-enabled systems, which excludes a number of potential users. Additionally, the performance gains are context-dependent; if expert reuse is low, the overhead of managing the cache may outweigh the benefits. Developers should also note that output can vary based on which experts are cached, leading to potential inconsistencies.

The bottom line

The new caching feature in llama.cpp offers a promising way to enhance performance for developers using MoE models on consumer GPUs. While it shows significant potential, it is still in development and has limitations that builders should consider. Keeping an eye on this PR could provide valuable insights for future projects.

FAQ

What is llama.cpp?

llama.cpp is an open-source project that provides tools for working with large AI models, particularly those utilizing Mixture of Experts architectures.

How does the caching feature improve performance?

The caching feature stores frequently used MoE experts in GPU memory, which speeds up token generation by reducing CPU load and improving resource efficiency.

Source: reddit.com