In quantitative biomedical research the binding constraint is hardly ever the classification algorithm: it is the protocol by which its performance is measured. WEKA — the Waikato Environment for Knowledge Analysis — is the Open Source tool that makes that protocol reproducible even for people without a computing background. That, more than its catalogue of classifiers, is why it appears with growing frequency in the literature on clinical and gene-expression data.

Context

WEKA comes out of the group of Ian H. Witten and Eibe Frank at the University of Waikato, New Zealand. The project started in the mid-1990s in Tcl/Tk, was then rewritten in Java, and since April 2000 the code has been hosted on SourceForge. It is distributed under the GNU General Public License. The current stable line is 3.2.x; alongside it runs the 3.3.x development line, where new classifiers and interfaces converge. The companion text is Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations by Witten and Frank (Morgan Kaufmann, 2000), which documents the implemented algorithms and provides the teaching base.

The tool has four interfaces. The Explorer is for interactive exploration of a dataset: tabular view, per-attribute histograms, scatter plots, a rendered decision tree. The Experimenter is for comparative experiments, with cross-validation and statistical tests. The command line automates. The Java API embeds the classes in one’s own applications. The input format is ARFF (Attribute-Relation File Format): a text file with a header that declares the relation name and the attribute types (@relation, @attribute, @data) and a CSV-style data section. A format that makes the types explicit is what makes a dataset exchange replicable across different groups.

Problem

The use case that brought WEKA into the clinical literature is classifying samples from many variables and few subjects. The canonical example is the Golub et al. leukaemia dataset, published in Science in 1999 (vol. 286, pp. 531–537): 72 tissue samples, 7129 gene-expression values from Affymetrix chips, a binary label between acute myeloid leukaemia (AML) and acute lymphoblastic leukaemia (ALL). Thousands of attributes and a few dozen observations: the opposite geometry to the one most classical statistical methods are tuned for.

On data of this shape the dominant risk is estimation optimism. With 7129 genes one almost always finds a subset that separates 72 samples perfectly, even when the label is pure noise. Dudoit, Fridlyand and Speed (Journal of the American Statistical Association, 2002) systematically compared several discrimination methods on Golub and two other microarray datasets: the accuracy estimate changes depending on whether variable selection happens inside or outside the cross-validation loop. Select the genes by looking at the whole dataset, then validate with cross-validation, and the numbers come out systematically too optimistic.

Architecture

WEKA gathers a catalogue of classifiers organised by family: decision trees with J48 (the Java implementation of Ross Quinlan’s C4.5), Id3 and REPTree; Bayesian classifiers with NaiveBayes and BayesNet; instance-based methods with IBk (k-nearest neighbours); functions with SMO — John Platt’s Sequential Minimal Optimization algorithm (Microsoft Research, 1998) for training Support Vector Machines — and Logistic; shallow neural networks with MultilayerPerceptron; meta-classifiers such as Bagging, AdaBoostM1 and Stacking; attribute-selection methods (InfoGainAttributeEval, CfsSubsetEval, wrapper approaches). For each one you get the trained model, per-class probabilities where they make sense, and metrics under configurable stratified cross-validation.

For clinical practice the length of this list does not matter. What matters is the FilteredClassifier class and the way the Experimenter chains pre-processing and classification: an attribute-selection filter wrapped around the classifier is re-fitted at every cross-validation fold, on that fold’s training data only. By construction, test-set information does not leak into the feature choice. The same scheme, hand-written in MATLAB or in R, is a recurring source of error; in WEKA it is the default behaviour when you compose the filter with the classifier, rather than filtering the dataset once.

The typical workflow in a biomedical lab is therefore this: pre-processing in the specialist tool (R/Bioconductor for microarrays, MATLAB for physiological signals, feature extraction from ROIs for images), export to ARFF, and finally a comparison of classifiers in WEKA under a single validation protocol shared by all of them.

Critical point

With the Experimenter you define a set of datasets and a set of classifiers with their hyperparameters, run N repetitions of cross-validation with the same random splits for every method, and apply a statistical test (a corrected paired t-test) to decide whether a difference in accuracy between two classifiers is significant or noise. On 72 samples the fold-to-fold variance is large, and a two- or three-point accuracy gap between J48 and SMO almost always falls inside the range where the test does not reject the null hypothesis.

The contribution that matters more than any algorithm is exactly this: making it routine, and all but free, to report whether a measured difference can be told apart from sampling noise. On small, high-dimensional clinical datasets it is what separates a result that survives an independent replication from one that does not.

Implications

For a clinical research group without a dedicated computing person, WEKA lowers the threshold for doing comparisons correctly: stratified cross-validation, feature selection wrapped inside the validation loop, and significance tests are done from the GUI, without writing the protocol code. The ARFF format ties WEKA to the pre-processing tools, and the Java API lets a trained model be embedded in an application: it serves both for exploration and for integration into a larger system. Several research projects build small interfaces on top of the WEKA API, specialised for a single disease.

Limits

The Java implementation, written for teaching clarity, does not hold up to datasets of millions of records; SMO and MultilayerPerceptron slow down markedly as size grows. The available networks are shallow multilayer perceptrons with classical backpropagation. WEKA expects the features to be already extracted: it does no representation learning, and all the extraction from signals or images stays upstream. Cost-sensitive learning for imbalanced classes — common in medicine, where the condition of interest is the minority — is there (CostSensitiveClassifier), but has to be set up by hand; and overall accuracy remains a misleading metric, to be replaced by AUC, precision and recall chosen case by case. For moderate-scale datasets with well-designed features, the most common situation in quantitative clinical research, these constraints are worked around by tending the protocol, not with compute.

Cover image: Screenshot of the “Weka GUI Chooser” launcher window: header with the software name and version, a photo of a weka bird, and four… — screenshot by University of Waikato, CC BY-SA 3.0 — https://commons.wikimedia.org/wiki/File:Weka_GUI_Chooser.png