On 4 January 2024 the StatNLP group at the Singapore University of Technology and Design released TinyLlama, a 1.1-billion-parameter model (1.1B), together with its training code, the datasets and the intermediate checkpoints, all under the Apache 2.0 licence (arXiv:2401.02385). For anyone working on governance the interesting aspect is not the parameter count: it is that the chain from raw data to final weights can be inspected step by step.

Context

Most of the models labelled “open” in 2023 release only the weights. Meta’s Llama 2 distributes its checkpoints under a licence that restricts commercial use and does not document, in reproducible form, how the pre-training corpus is composed. Mistral 7B releases its weights under Apache 2.0, but not the dataset or the training code. In both cases a third party can run and fine-tune the model, yet cannot answer the question that decides an audit: with what data and by what procedure are these numbers obtained?

TinyLlama sits at a different level of openness. The repository publishes the data-processing recipes, the training scripts and the checkpoints saved at regular intervals during training, which the abstract describes as “around 1 trillion tokens for up to 3 epochs” — on the order of 3 trillion tokens in total. At that point the distance between “downloadable weights” and “reconstructable pipeline” can be verified rather than taken on trust.

Architecture

The model reuses the Llama 2 architecture and tokeniser: RoPE for positional encoding, RMSNorm in a pre-norm configuration, SwiGLU activation, and grouped-query attention with 32 query heads and 4 shared key-value groups. Compatibility with Llama 2 has one precise practical consequence: every tool in the Llama ecosystem — quantisation, inference, fine-tuning libraries — runs without adaptation.

Training brings together two public corpora. SlimPajama, a deduplicated and cleaned version of RedPajama, supplies the natural-language text; StarCoderData supplies the code. The mix follows a natural-language to code ratio of 7:3, for a stated total of roughly 1 trillion unique tokens repeated over up to 3 epochs, on the order of 3 trillion in total. Training runs on 16 A100-40G GPUs over about 90 days, with a reported throughput of 24,000 tokens per second per GPU and a model FLOPs utilisation of 56% without activation checkpointing (arXiv:2401.02385, section 2). To fit within 40 GB of GPU memory with a 16k-token per-device batch, the authors use Flash Attention 2, PyTorch Fully Sharded Data Parallel and the fused SwiGLU operators from xFormers.

The critical point

The stated research question concerns the Chinchilla scaling laws (Hoffmann et al., 2022), which for a given compute budget indicate an optimal ratio of about 20 training tokens per parameter. At 1.1 billion parameters, compute-optimality would sit around 22 billion tokens. TinyLlama uses roughly 140 times that. The published benchmarks show that performance keeps improving even in the late phase of training, well past the point Chinchilla predicts.

On deployment the sums add up. The Chinchilla scaling laws minimise training cost at fixed compute, but a model is trained once and then run many times. Training a smaller model for longer shifts the cost from training to inference, where it pays off if the model has to run on constrained hardware such as edge or mobile. The verifiable quantity here is the benchmark curve as a function of tokens seen, not a qualitative judgement.

For governance the point lies elsewhere. With the intermediate checkpoints public, the trajectory of the metrics across training stays inspectable by anyone, without re-running 90 days of compute on 16 A100s. A reviewer can check whether the stated curve matches the released checkpoints: a documented claim thus becomes a reproducible one.

Implications

The definitions of “open AI” circulating in 2023 remain ambiguous. The Open Source Initiative began a public process in late 2022 towards a definition of Open Source AI, still under way as of this date, precisely because applying the Open Source Definition to models forces a decision on what must be available beyond the weights: the training code, the data, or both. Three components that can carry entirely independent licences and availability.

A release that includes datasets, code and intermediate checkpoints under Apache 2.0 satisfies the most demanding of the cases under discussion. It is a property that does not depend on scale: it holds for a 1B model as for a 70B one, and on a small scale the economic barrier to actual reproduction — not just inspection — is far lower. Retraining a 1.1B model on public corpora is within reach of an academic lab; re-running a training run of hundreds of billions of parameters is not. On small models, stated verifiability and practicable verifiability coincide more readily.

Limits

Opening the pipeline does not resolve upstream data provenance. SlimPajama derives from RedPajama, itself a replica of web corpora; tracing the lawful basis for using each source document remains beyond the reach of the release. The Apache 2.0 licence covers code and weights; it says nothing about the rights in the training content.

The reported benchmarks measure capability on standard language understanding and generation tasks; they say nothing about adversarial robustness, behaviour on out-of-distribution input, or systematic bias, which call for separate evaluation. And a 1.1B model remains bound by the capacity limits of its scale: pipeline reproducibility is orthogonal to output quality. The release makes verifiable how the model was built, not how well suited it is to a given downstream task.

Cover image: NVIDIA Tesla A100 data-center GPU board seen from the front, with an elongated black body and aluminium heatsink — photo by Nvidia, CC BY-SA 4.0 — https://commons.wikimedia.org/wiki/File:Nvidia_Tesla_A100.png