Inference: Chinese engine boosts DeepSeek V4.1 Flash throughput 6.87x on PCIe GPUs

The debate over AI chips is shifting. It is no longer just about who can buy the fastest hardware, but who can squeeze the most performance from what they already have. That is the argument METASTONE, a Chinese company, makes through its Meta-Infer inference engine, according to 量子位 (QbitAI).

The gap between framework and hardware

Many graphics cards can load a model, download weights, and start a service. They "work." Under load testing, however, real performance often falls far short of advertised levels. The model is not faulty, and the hardware is not broken. The loss stems from the gap between the framework and the hardware. These cards lack high-speed interconnects and do not appear in the official validation matrices of open frameworks. As a result, operators silently fall back on generic, inefficient implementations. Communication parameters remain calibrated for other architectures, and memory or parallelism settings keep default values meant for different hardware.

6.87 times the throughput

Meta-Infer attacks this problem in software alone. It does not change the model structure or task semantics. The approach has four parts: kernel completion, operator optimization and communication rewrite, tuning of parallelism and memory capacity, and cache reuse.

The first part fixes metadata and page sizes that block native optimized operators. It replaces outdated kernels and widens the threshold for fast communication paths. On an environment of eight PCIe cards, using the reference community version for DeepSeek-V4.1-Flash, input throughput was only 1,932 tokens per second. After this correction phase, it rises to 5,850 tokens per second.

The second part tackles bottlenecks. On a PCIe architecture, bandwidth between cards is far lower than on a dedicated interconnect. Default strategies waste time on communication waits. The engine fuses operators, overlaps computation with communication, and rewrites collective communication logic. It reaches 13,274 tokens per second, a factor of 6.87, with support for very long context.

The methodology is generic. On another model, input throughput goes from 14,546 to 22,584 tokens per second. From the co-adaptation phase alone, several models gain between 20% and 33% in throughput.

A third-party operator

METASTONE presents itself as an independent end-to-end computing operator, not tied to any model vendor. The company says it manages more than 20,000 petaflops of resources. It runs about ten intelligent computing centers and two national supercomputing centers, with a commitment to availability above 99.95%. Its team claims three Gordon Bell prizes.

When hardware is constrained, software optimization becomes the main lever for performance.