Open weights is not open source

There is a published three-part test for open source AI. Nearly every model marketed as open passes one part of it: the weights are downloadable.

Part of the Trust and evals track on lAItest.

Almost every model described as open source is not open source. The weights are downloadable, which is a different claim.

There is a published test, and it has three parts.

The three-part test

The Open Source Initiative published a definition for AI systems. To qualify, three things are required: information about the training data sufficient to rebuild a substantially equivalent system; the complete training code and the inference code; and the model parameters, under terms the definition approves. On top of that, the usual freedoms to use, study, modify and share it for any purpose. Nearly every model marketed as open publishes the third item and stops there.

Downloadable, or not, across 36 tracked models

Try it

Filter to models whose weights you can download, then look at what those same models cost to rent from their makers anyway. This step is an interactive widget; open the lesson to use it.

A common misconception

Commonly believed: Open weights means I downloaded it, so I can do what I like with it.

Actually: Downloadable, permissive and reproducible are three separate questions. Some open-weight models carry MIT or Apache 2.0 terms and really are unrestricted. Others carry custom licences with conditions: attribution inside your interface, or a revenue threshold above which you must negotiate a separate agreement with the maker. And a licence that permits everything still does not let you rebuild the model, because the data and the training code were never published.

In one sentence

Open weights is a claim about what you can download. Open source is a claim about what you could rebuild. Almost everything on offer is the first one.