On 9 February 2022 the EleutherAI collective made the weights of GPT-NeoX-20B downloadable, a 20-billion-parameter autoregressive language model, under the Apache 2.0 licence and with the training code published separately. The date is borne out by EleutherAI’s blog announcement and the appearance of the files on the The Eye mirror. For anyone working on software governance, the interesting part is not the size of the model but the package shipped with it: weights, code, a documented corpus and a permissive licence, all together.

Context

For most of 2020 and 2021, language models in the same class as GPT-3 were reachable almost only through commercial APIs. You query a remote endpoint, you pay per token, and neither the weights nor the training data are inspectable. That distribution model has concrete consequences: anyone studying the system’s behaviour sees only the outputs, not the parameters; anyone wanting to reproduce a result cannot; anyone needing to assess bias or training-data memorisation works blind.

EleutherAI, a collective formed in 2020 with the stated aim of openly replicating GPT-3-like models, had already released GPT-J-6B on 9 June 2021, also under Apache 2.0. With GPT-NeoX-20B it takes the same approach to a larger scale.

How the release is put together

The model was trained with gpt-neox, the training library EleutherAI maintains on GitHub. The library rests on NVIDIA’s Megatron Language Model and integrates DeepSpeed techniques, ZeRO included, for parallelism across multiple GPUs. The code ships under Apache 2.0: the parts derived from NVIDIA keep the original copyright headers, while the collective’s own contributions are attributed to EleutherAI. The training hardware was provided by CoreWeave, and the announcement describes the configuration used.

The training data is The Pile, an English-language text corpus of roughly 825 GiB curated by EleutherAI and described in the December 2020 paper by Leo Gao and colleagues (arXiv:2101.00027). The Pile brings together 22 heterogeneous subsets — academic papers, books, code, filtered web text — and from January 2022 it has a dedicated datasheet (arXiv:2201.07311) documenting its provenance, composition and known limitations, along the lines proposed by Gebru and colleagues for machine-learning datasets.

Three artefacts, then, shipped together:

  • the model weights, downloadable under Apache 2.0;
  • the training code, with configurations that describe the architecture and hyperparameters;
  • the corpus documentation, as a paper and a datasheet.

The critical point: permissive licence and release responsibility

Apache 2.0 sets no use restrictions. No clause forbidding specific applications, no case-by-case approval, no access gate. Anyone who downloads the weights can use, modify and redistribute them, including commercially, as long as they meet the attribution obligations and keep the licence notice. This is a governance choice different from distributing models behind an API with server-side enforceable terms, and different too from licences that carry acceptable-use clauses.

The flip side is that the responsibility for use slides entirely downstream, onto whoever downloads. EleutherAI says so plainly in the announcement: the models are “research artifacts” and the authors write that “we do not recommend deploying either in a production setting without careful consideration”, urging readers to consult the paper and the datasheet on the corpus before use. The datasheet exists precisely for this: making the composition of the training set inspectable is the technical precondition for assessing bias, contamination and memorisation risk — an assessment that a closed system, from the outside, does not allow.

There is a governance tension here worth naming. A permissive licence pushes scientific reproducibility and inspectability to the limit — properties a paid endpoint does not give — while giving up any contractual leverage over downstream use. Publishing weights and a datasheet makes independent verification possible; it does not make the model safer in itself. These are two planes to keep apart when discussing “open” as applied to models.

What changes for reproducibility

For researchers, having weights, code and a documented corpus together moves the boundary of what can be verified. With a model behind an API you compare outputs; with public weights you inspect parameters, run the model locally, fine-tune it, and measure memorisation of the training set by probing the network directly. With the training code and the configurations, in principle you reconstruct the training process — the limit is compute cost, not access.

The datasheet adds the missing link. Without a structured description of the corpus, even with the weights in hand, the analysis of a model’s behaviour is left without the context of the data that produced it. It is the difference between noting that a model emits a certain output and being able to trace it back to the corpus subset that makes it likely.

Limits

Reproducibility “in principle” is not reproducibility in practice. Training a 20-billion-parameter model takes a GPU cluster that most academic research groups do not have; publishing the code removes the barrier of software access, not that of compute cost. The announcement itself calls the software a work in progress and flags open bugs and inefficiencies.

“Open” is not a binary property either. Apache 2.0 weights, training code and a datasheet are a high degree of openness, but questions remain that the release does not settle: raw data gathered from third-party sources is subject to the rights of the original holders, and corpus documentation, however thorough, is not equivalent to a full provenance audit of every document. Assessing an “open” model means looking at which artefacts ship and on what terms, without stopping at the label.

Cover image: Logo of the EleutherAI collective: a stylized icon on a dark square background — logo by Logo disegnato da Sid Black; versione SVG vettorizzata da VulcanSphere, CC0 — https://commons.wikimedia.org/wiki/File:EleutherAI_logo.svg