Classifying sleep stages from biometric signals

Staging sleep is a multichannel time-series classification problem where the labels are set by clinical rules, not by statistical consensus. The reference standard is the American Academy of Sleep Medicine (AASM) scoring manual: it assigns each 30-second epoch one of five labels (W, N1, N2, N3, REM), following explicit morphological criteria over the electroencephalogram (EEG), the electrooculogram (EOG) and the chin electromyogram (EMG). Anyone building an automatic classifier therefore works against a regulated ground truth, and the whole validation argument follows from that.

The problem, in concrete terms

Polysomnography (PSG) records several physiological signals in parallel through the night: cortical activity (EEG, usually frontal, central and occipital derivations), eye movements (EOG), muscle tone (submental EMG), and almost always respiration, oxygen saturation, body position and ECG. The scorer divides the recording into 30-second epochs and assigns each a stage following the AASM manual, currently at version 2.3 (2016). The rules are operational: N2, for instance, requires K-complexes or sleep spindles when N3 criteria are not met; N3 is defined by a share of slow-wave activity (0.5–2 Hz, amplitude above 75 µV) covering at least 20% of the epoch.

Two technical consequences follow. First, the label is not a physical truth, it is the output of a protocol. Agreement between scorers on human PSG sits around 80–83% epoch-by-epoch concordance, and that is the ceiling beyond which an automatic classifier cannot be validated against a single scorer without absorbing the noise of the label. Second, the classes are heavily imbalanced — N2 dominates, N1 is a minority and ill-defined — which makes raw accuracy a weak metric and forces the use of macro F1 and per-stage confusion matrices.

Signals, formats and available data

Exchange of PSG recordings rests on a stable, dated standard: the European Data Format (EDF), from 1992, and its EDF+ extension from 2003 (Kemp & Olivan, Clinical Neurophysiology 114), which adds time-stamped annotations, discontinuous recordings and hypnograms encoded as an annotation channel. A typical PSG dataset is therefore a pair: the signal in EDF, the reference hypnogram in EDF+.

Among public data the reference is the Sleep-EDF Database [Expanded] on PhysioNet, brought in 2013 to 61 nights of PSG with hypnograms (EEG Fpz-Cz and Pz-Oz, horizontal EOG, chin EMG, sampled at 100 Hz). By machine-learning standards it is a small corpus, and this is the constraint that weighs most: what decides how much can really be validated is the scarcity of annotated recordings, not model capacity. Anyone working on top of it soon has to strike agreements with sleep centres to get real clinical material.

Two families of approaches

The field splits into two traditions that measure different things.

Actigraphy estimates sleep and wake from wrist movement alone: an accelerometer produces activity counts per epoch. The historical algorithms — Cole-Kripke (1992) and Sadeh (1994) — combine the counts linearly, with weights, over a time window, then apply re-scoring rules. They are robust and validated, but with a known structural limit: high sensitivity to detect sleep (0.88–0.96) and low specificity for wake (0.35–0.64). In practice, someone awake but lying still ends up classified as asleep. And actigraphy does not distinguish stages: it separates sleep from wake, nothing more.

PSG-based staging tackles the full five-class problem. Here, by late 2016, machine learning had reached results in line with agreement between humans: Tsinalis et al. (arXiv:1610.01683, October 2016) classify stages from single-channel EEG with convolutional networks, report a mean F1 around 81% and — this is the interesting point — observe that the learned filters correspond to morphological criteria in the AASM manual. The network, trained only on labels, rediscovers the features that scorers use explicitly. Less a matter of the model being clever than a clue: the information useful for staging sits exactly where the clinic places it.

The critical point: validating against a noisy ground truth

The difficulty is not training a classifier, it is demonstrating its reliability. Three constraints are entangled.

First, the ground-truth ceiling: if two scorers agree at 80%, a classifier agreeing at 85% with a single scorer is not necessarily better — it may have learned that scorer’s idiosyncrasies. Correct validation requires consensus scoring from multiple readers, which in public datasets is costly and rare.

Second, generalisation across montages and populations: different EEG derivations, different hardware, different age bands shift the signal distribution. A model that runs on Sleep-EDF does not automatically run on clinical recordings with a full AASM montage. The test that counts is cross-dataset validation, and it is precisely the one most papers published in early 2017 do not report.

Third, imbalance: N1 has few epochs and is the class humans agree on least. An overall accuracy of 74% can hide an F1 below 0.45 on N1. Without the per-class breakdown, the aggregate figure says little.

What changes on the regulatory side in Europe

In Europe a software that produces output supporting the clinical assessment of sleep falls under the Medical Device Directive 93/42/EEC. The risk class depends on the declared intended use. Software with an informative or supporting function, without a measuring function or sterile conditions, normally falls in Class I: there, conformity assessment is the manufacturer’s direct responsibility and does not bring in a Notified Body — though the technical file remains mandatory, along with risk management per EN ISO 14971, the software life cycle per IEC 62304 and clinical evaluation per MEDDEV 2.7.1 Rev. 4 (June 2016). CE marking in Class I is a self-declaration, but the validation documentation behind it is exactly the critical point above: without a traceable chain from dataset to per-class metric to comparison against consensus scoring, the technical file does not stand.

Limits

The above is a snapshot of the state of practice in early 2017 and leaves two questions open. The first is the move from laboratory EEG to low-cost wearable sensors (accelerometer, photoplethysmography, heart rate), where the signal is poorer and actigraphy remains the weak reference available. The second is the absence of a large public benchmark with consensus scoring from multiple scorers: until one exists, any accuracy figure should be read as relative to the scorer and the montage of the dataset on which it was measured, not as an absolute property of the model.

Around these constraints — staging from polysomnographic and actigraphic time series, the path toward EU Class I medical device certification — the startup SleepActa is founded, whose formal incorporation noze documented in an insight: https://www.noze.it/en/insights/sleepacta-founding/.

Cover image: Patient fitted with scalp electrodes and connecting wires for a polysomnography (sleep study) recording — photo by Robert Lawton, CC BY-SA 2.5 — https://commons.wikimedia.org/wiki/File:Pediatric_polysomnogram.jpg