An 8 GB Raspberry Pi 5 now runs a quantised language model with no network connection at all, at a speed that suffices for classification and extraction tasks. The point is not throughput: the text being processed never crosses a network boundary, and so falls outside the obligations of whoever governs transfers of personal data.

Context

In April 2024, within a few days of each other, two models came out that widen the range of what runs on hardware costing a few tens of euros. Meta released Llama 3 8B on 18 April; Microsoft published Phi-3 mini, 3.8 billion parameters, on 23 April, describing it as small enough to run locally on a phone (Microsoft Research, Phi-3 Technical Report). Both ship in GGUF, the container llama.cpp uses for quantised weights.

Quantisation is the lever that makes execution on an ARM CPU viable. A 16-bit weight takes four times the space of a 4-bit one; the Q4_K_M scheme in llama.cpp brings weights down to roughly 4.5 bits per parameter with mixed precision — the more sensitive tensors stay at higher resolution (llama.cpp quantize documentation). Phi-3 mini at Q4_K_M takes about 2.3 GB: it fits comfortably within a Pi 5’s 8 GB and leaves room for the operating system and the context cache.

The question that opens here is one of data governance, not performance. An assistant that sends text to an external endpoint transfers that text to a processor, with the contractual consequences and — under the GDPR — the lawful-basis and minimisation obligations and, where the endpoint sits outside the European Economic Area, the international-transfer obligation. Keeping inference on the device cuts the transfer at the root, not as a mitigation measure but as a physical fact.

Architecture

The Raspberry Pi 5 (October 2023) carries a Broadcom BCM2712 with four Cortex-A76 cores at 2.4 GHz, ARMv8.2 with the dotprod extension, and up to 8 GB of LPDDR4X-4267 (Raspberry Pi specifications). The cores implement the NEON instructions and the 8-bit dot product that llama.cpp draws on in its matrix-multiply kernels: that is where most of the CPU speed comes from, with no dedicated GPU.

The minimal stack is three pieces. The runtime is llama.cpp or Ollama, which wraps llama.cpp behind an HTTP server and more convenient model management. For sustained inference, llama.cpp built from source with the NEON and dotprod flags stays about 10-20% faster than Ollama; Ollama is handier for quick prototyping. The model is a quantised GGUF pulled from Hugging Face. The operating system is 64-bit Raspberry Pi OS — the March 2024 release improved inference performance considerably over earlier builds (aidatatools benchmark).

Here are the measured speeds on an 8 GB Pi 5, CPU only, which set what is realistic:

  • 1-2B models at Q4: tens of tokens per second, usable in real time;
  • Phi-3 mini 3.8B at Q4_K_M: roughly 4-6 tokens per second, acceptable for short answers and structured tasks;
  • 7-8B models at Q4: below 2 tokens per second, under the threshold of interactive usability (independent benchmarks on the 8 GB Pi 5).

These numbers should be read per task, not per chat. At 4-6 tokens per second a 3.8B model classifies an email, extracts the fields from a document, routes a request or summarises a paragraph within acceptable service times. It does not hold up a long conversation, and it does not need to.

The critical point

The constraint that dominates is memory, not compute. The model has to fit in RAM alongside the context cache, which grows with window length; the Pi 5’s 8 GB set a practical ceiling around 4B parameters at Q4, if you want a non-trivial context and headroom for the rest of the system. The 4 GB board only makes sense for the smallest models and short contexts.

A design choice follows from this: the Pi is not the place to run a large general-purpose model, but the place to run the smallest model that solves a precise task. On academic benchmarks Phi-3 mini scores comparably to Mixtral 8x7B and GPT-3.5 (Microsoft Research, Phi-3 Technical Report) — a claim to verify on your own domain, because generic benchmarks predict performance on a narrow task poorly. Validation is done on the real data of the use case, not on published scores.

There is also a hidden cost: Q4 quantisation degrades quality relative to the full-precision model, by an amount that depends on the task. For classification and extraction the drop is often negligible; for multi-step reasoning it shows more. It is to be measured, not assumed.

Implications

Moving inference onto the device changes the compliance profile even before the code. Under the GDPR, processing that happens entirely locally generates no transfer to a third party and, where there is no connectivity, no international transfer: two of the most expensive items in a risk assessment disappear by construction. Everything else remains — lawful basis, notice, minimisation, data-subject rights — but the boundary of the processing coincides with a physical object you can inspect.

The scenarios where this weighs are those in which data cannot leave by constraint rather than by preference: medical and legal practices, devices in environments without a reliable network, sensors producing text that should not be centralised, document archives you want to search conversationally without exposing them. The same Pi can host the full offline pipeline — whisper.cpp for transcription, a small model for understanding, Piper for speech synthesis — without anything reaching a server.

Limits

This configuration holds for narrow, low-volume tasks. It does not replace GPU inference for medium models or for concurrent load: a single Pi serves one user at a time at a few tokens per second. First-token latency on a 3.8B model is a few seconds, acceptable for an asynchronous interaction and irritating for a synchronous one. And the fact that data stays local does not make the system compliant on its own: it removes one category of risk, not the duty to govern the others. Checking that the quantised model does its job on the real domain remains the part of the work no published benchmark can stand in for.

Cover image: Top-down view of a Raspberry Pi 5 single-board computer: green printed circuit board with the square SoC, two ribbon-cable… — photo by SimonWaldherr, CC BY 4.0 — https://commons.wikimedia.org/wiki/File:Raspberry_Pi_5.jpg