OpenCV is a general-purpose computer vision library, written for robotics and industrial vision, and for that very reason it works well on a well-bounded slice of medical imaging: photographic-quality 2D images. The library — the Open Source Computer Vision Library, started inside Intel Research in 1999-2000 on Gary Bradski’s initiative, first stable 1.0 release in October 2006, BSD licence — contains nothing specific to medicine. Yet it shows up more and more often in the biomedical literature of 2007-2008, because the basic operations it provides are the ones needed before any classification: filtering, segmentation, extraction of numeric descriptors.

I am writing this note after a few months of work with the 1.x line on fundus images and microscopy smears, in a research setting rather than a product one. That distinction changes everything, and I come back to it at the end.

Context and application domain

The reference for medical imaging remains ITK (the Insight Segmentation and Registration Toolkit), begun in 1999 under a U.S. National Library of Medicine contract around the Visible Human Project. ITK is built for 3D volumes, for the registration and segmentation of CT and MR, and its templated C++ pipeline model reflects that purpose. OpenCV occupies different ground: 2D, photographic-type images, where each pixel is a sample on a regular grid and the analysis does not ask you to reason about a volume.

A good share of diagnostic modalities produces exactly this kind of input. Digital mammography is a 2D projection. Retinography and fundus photography are colour images. Dermoscopy, optical microscopy, endoscopy, digital pathology slides: all 2D, all treatable with the same operators OpenCV applies to any ordinary photograph. For these domains the library has a ready-made repertoire; for CT and MR volumes it is the wrong tool, and pushing it that way means reimplementing what ITK already does.

What the 1.x line offers

Version 1.0 is a C library, with the IplImage data structure inherited from the Intel Image Processing Library acting as the image container. A C++ rewrite around a new cv::Mat structure is under way, but the code one writes in production today is C with those conventions. The repertoire useful for biomedical images falls into a few blocks.

Pre-processing and filtering: colour-space conversions (RGB, HSV, greyscale), histograms and equalisation, Gaussian, median and bilateral filters, Sobel and Laplacian gradients, Canny edge detection. On a mammogram they flatten the background and bring out microcalcifications; on a fundus image they normalise the uneven illumination typical of acquisition.

Morphological operations: erosion, dilation, opening, closing, top-hat, watershed. These are the daily bread of segmenting structures against a noisy background — separating adjacent cell nuclei, cleaning up a binary mask, isolating a pigmented lesion from the surrounding skin.

Segmentation and shape analysis: global, adaptive and Otsu thresholding; contour extraction and approximation; connected components; the Hough transform for lines and circles, handy for locating the optic disc or tracing straight vascular structures. From a contour you derive moments, area, perimeter, shape descriptors — the raw numbers a classifier then runs on.

Feature detection: Harris and Shi-Tomasi corners (cvGoodFeaturesToTrack), the FAST detector, MSER regions. One concrete constraint belongs here: SIFT (Lowe, 2004) and SURF (Bay, 2006), the descriptors most used for robust matching, are patent-encumbered. They can be used in research, but anyone thinking about a product has to account for them from day one.

Classification: the machine learning module includes SVM (a libsvm wrapper), k-nearest neighbours, AdaBoost-boosted classifiers, Random Trees, Normal Bayes, shallow multilayer perceptrons and Expectation-Maximization. That is enough for many pilot studies without leaving the library.

Critical point: the Viola-Jones detector

The OpenCV element with the most impact on medical imaging is the Viola-Jones cascade detector, described in 2001 and available out of the box through cvHaarDetectObjects. It works like this: a window slides over the image at several scales, and at each position it passes through a sequence of stages; each stage evaluates Haar-like features — sums and differences of light and dark rectangles, computed in constant time thanks to the integral image — and if a stage rejects the window, that window is discarded at once. Most windows fall at the early stages, and this is what makes the detector fast.

OpenCV ships pre-trained cascades for faces, but for medicine the face is not the point: the point is the training pipeline supplied with the library, with which you train your own cascade on positive and negative examples. People have used it to attempt detection of microcalcifications in mammograms, cells in a slide, regions of interest in repetitive images. The method has a well-known structural limit: it holds up well on frontal, low-variability patterns and loses effectiveness quickly on objects with strong variation in shape, scale and orientation — the typical condition of real lesions. It is a reasonable starting point, not a solution that transfers without work.

The typical workflow

A clinical research project that adopts OpenCV almost always follows the same path. The image arrives in its native format — DICOM, TIFF, PNG — and has to be loaded: OpenCV does not read DICOM, so loading goes through a dedicated library such as DCMTK or GDCM, and the pixels are then copied into an IplImage. Pre-processing follows with the library’s operators, then extraction of numeric features (moments, texture, shape and colour descriptors). Classification is done inside OpenCV or by exporting the features to WEKA or R, which have richer statistical tooling. It closes with validation: cross-validation, ROC curves, comparison against the reference ground truth annotated by the clinician.

These months are also changing the language this work gets written in. The library’s Python bindings, with NumPy alongside, are becoming a genuine alternative to MATLAB for prototyping in academic biomedical research: the same vectorised logic, more readable code, no licence cost. For now it is an emerging option, with bindings not always complete against the C API.

Limits

OpenCV has sharp boundaries when carried into medicine, and it is better to state them before starting a project than after.

DICOM is not native: an external parser is always needed, and with it you have to handle the correct conversion of pixel values (rescale slope/intercept, windowing) that OpenCV ignores entirely. 3D volumes are out of scope: for CT and MR the tool is ITK. Numeric precision is not the library’s priority, performance-oriented from birth; delicate operations such as high-quality interpolation or sub-pixel registration ask for extra care and sometimes a different library.

The limit that weighs most is not technical. The BSD licence allows OpenCV to be used in a commercial product, but using such a component inside a medical device is another matter: qualifying the software as part of a regulated system entails a documented lifecycle and risk management to be planned from day one, not stuck on at the end. A general-purpose library, developed by a community with no obligation toward the clinical domain, carries none of these guarantees: whoever integrates it has to put them in.

And then there is data variability, which no library solves. Different acquisition protocols, different scanners, uncontrolled illumination in dermoscopy and endoscopy: a method that runs well on one centre’s dataset often does not generalise to another’s. It is not a flaw in OpenCV, it is the starting condition one works with, and the library neither hides nor worsens it — it leaves it whole for the researcher.

For anyone starting a study today on non-volumetric 2D images, OpenCV 1.x is the most mature and best-documented open source tool available, with a transition toward C++ already visible on the horizon. Staying on the stable 1.x line or waiting for the rewrite is a trade-off between what runs today and what will run better tomorrow.

Cover image: Fundus photograph of a healthy right eye: a reddish-orange circle showing the bright optic disc on the right, the macula at the… — photo by Mikael Häggström, CC0 — https://commons.wikimedia.org/wiki/File:Fundus_photograph_of_normal_right_eye.jpg