Open weights are closing the gap, but the gap resets every release cycle

The report is version 1.1 of a recurring assessment Mozilla first published on its blog on 14 July. According to Tom's Hardware, it is built on a Mozilla and SlashData survey of roughly 1,400 developers, OpenRouter traffic data and third-party benchmark indices. Mozilla is a nonprofit advocate for open models, and TIME reported on 14 July that Mozilla CTO Raffi Krikorian described the report as partly advocacy. That framing matters when reading the numbers.

Mozilla's central claim is that the best open model trailed the closed leader on the Artificial Analysis Intelligence Index by three points, at 60% of the price, and sat two points behind Claude Fable 5 at 30% of the price. Fitting METR task-horizon data, Mozilla puts the open-closed gap at around 4.4 months, in line with Epoch AI's four-month estimate. METR is a research nonprofit that scores models by the length of task, in human working time, that they complete half the time.

What the four months actually measures

By Mozilla's fitted estimate, closed models handle tasks that take human experts 8 to 12 hours. Open models reach that level about four months later. Mozilla computes open capability doubling every 3.9 months against 5.5 months for closed models. On vals.ai's Terminal-Bench 2.1, which runs every model through the same harness, Z.ai's GLM-5.2 scored within a point of Claude Opus 4.7 and about four points behind Opus 4.8, at less than one-fifth the cost per test.

"We see the decision to pay for closed [models] as workload-specific rather than organization-specific," Krikorian told Ars Technica in an email.

Adoption on the developer side looks different from revenue. On OpenRouter, a marketplace that routes traffic to hundreds of models, Mozilla counted eight of the top ten models by August token volume as open weights, seven of them Chinese-built. Closed providers nonetheless took 96% of model-layer revenue on OpenRouter from May to September 2025, according to the Linux Foundation. Free or cheap tokens and paid revenue are not the same market.

Mozilla counts 16 notable open releases, but none delivers the data recipe required by the Open Source Initiative's definition. That distinction is the whole argument of a 19 September OSI post titled "Open Weights Are Good. Open Source Is Better." Open weights, the OSI writes, let you run a model locally, pay a third party to host it, or fine-tune it on your own data. They do not let you fully study it, because the code and the training data or a detailed account of how the data was built are not part of the release.

The hardware caveat

The 4.4-month figure and the 30% token price are measured API to API on hosted endpoints at list price. Mozilla's own hardware chart tells a harsher story. The best open model that fits one server scores 52.6; the best that fits one GPU scores 40. The drop from the top is 10 and 23 points respectively, a larger gap than four months implies.

Kimi K3's native MXFP4 checkpoint runs about 1.56TB across 96 shards, per the report. Mozilla's serving configuration lists 64 or more accelerators, while vLLM calls for at least eight GB300 GPUs, with multiple nodes for production traffic. Tom's Hardware put the memory need near 1.5TB in July. The report describes this as open but not runnable by most who hold it. One exception it names is Thinking Machines' Inkling-Small, under Apache 2.0, whose NVFP4 version fits one B300 at a 180GB floor.

There is also a timing problem. The report's data stops at 1 September. Since then, Artificial Analysis has moved its index to v4.3 with a different evaluation set. The live board has Claude Fable 5.1 at 53 on its highest effort setting with Kimi K3 at 44, numbers Mozilla did not plot and which are not comparable to the v4.1.1 figures it used. vals.ai's Terminal-Bench 2.1 board, updated 11 September, is now led by GPT-6 Astra at 87.27%, with Fable 5.1 at 85.02%. Mozilla's own chart caption reads: "the gap resets every release cycle."

K3 also carries an allegation detailed in the 8 September NSA, CISA and FBI joint advisory AA26-251A. The claim, which Mozilla's report states as "asserted, and unshown," is that Moonshot extracted Claude Fable 5 data to train K3 through distillation, the practice of training one model on another model's outputs. On 17 July, Artificial Analysis had K3 at 57 against Fable 5's 60; on 1 September, Mozilla had it two points back.

Small models, narrow jobs

Not every open-weight release is chasing the frontier. Superwhisper's S1-mini, published on Hugging Face, is a 0.6B-parameter text normalizer for speech-to-text output. It takes a raw ASR transcript and rewrites it as written text: fillers removed, false starts and self-corrections resolved, punctuation and capitalization applied, spoken numbers, dates, times, currency and email addresses rendered in written form. On a held-out set of 7,519 English cases it reaches 94.8% token accuracy, and the quantized build is a 462 MiB file that runs on a laptop CPU.

The model card is unusually blunt about what it is not. S1-mini is not a chat model and will not follow general instructions. It is fine-tuned from Qwen/Qwen3-0.6B and expects a control line at the top of the input specifying styling, structure and context. Skip the system prompt, change its wording, or send values outside the trained sets, the card warns, and the model can hallucinate or produce garbled output. It is English only, licensed Apache 2.0 plus a naming clause. The Hub sidebar reports 0.8B parameters, but the card explains that model.safetensors stores lm_head.weight as a materialized copy of the input embedding, counting the 155.6M-parameter embedding twice: 751.6M tensor elements against 596.0M unique parameters. The layout is inherited from Qwen3-0.6B, which reports 0.8B for the same reason.

A similar pattern shows up in a 22 September deployment write-up by James Cruce. He chose Laya, a 421M-parameter open-weight decision model, over TypeSafe's hosted Jev for the small, bounded choices his coding agents make. The checkpoint, convaiinnovations/laya-typed-decisions, has a ModernBERT-large encoder and a 1,024-token limit, and returns typed answers: a choice among named options, a score on an ordered scale, or a probability for a yes-or-no statement. No paragraphs to parse. Jev, by contrast, is an early-access hosted service priced at $0.042 per million input tokens with no output-token charge, and TypeSafe says it handles choices with up to 255 options.

Cruce's stated reason for the switch was not cost. He wanted routine decisions running on his Mac Studio with local data staying local, and he wanted control over the exact model, the permitted options and the failure behavior. He did not calculate a break-even and says he has no savings number to give. He also flags limits: the publisher trained Laya on four synthetic workflows, and model routing is not one of them, so the result is one deployment on an M3 Ultra with 256GB of unified memory, not a general verdict on Laya against Jev. The model card warns the model is overconfident and needs domain-specific calibration. "A typed answer can be structurally perfect and factually wrong," he writes.

The legal picture around weights is just as unsettled. A paper by A. Feder Cooper, Mark A. Lemley and co-authors, first submitted on 18 May 2025 and revised through 20 July 2026, measured memorization across 200 books and 14 open-weight models in over 3,000 experiments. Its finding cuts against both sides of the copyright fights. Most models do not memorize most books, in whole or in part. But Llama 3.1 70B entirely memorizes some, including Harry Potter and the Sorcerer's Stone, to the point that the whole book can be extracted almost verbatim from the first few words as a prompt. The authors write that their results have significant implications for copyright cases, though not ones that unambiguously favor either side.

None of this settles whether open weights win. The metrics that look best for openness, price per token and share of routed traffic, sit next to metrics that look worse, single-server and single-GPU capability and revenue share. Mozilla's own caption is the honest summary: the gap resets every release cycle.