In an economic and financial analysis domain the useful knowledge is scattered across research notes, model spreadsheets, methodological memos and internal correspondence, and the operational problem is not storing it but finding it again when it is needed. On systems of this kind, for Prometeia, I saw at close range how today’s open-source stack addresses the question and where it instead leaves the integration work exposed.

Context and problem

A firm doing risk analysis and budget forecasting produces information with three properties that are awkward for a document system.

It is heterogeneous: discursive text (research notes, scenario commentary), tabular data (exposures, time series) and binary artefacts (spreadsheets, presentations, report PDFs) all coexist. An index that handles only the text loses half the value.

It is versioned in practice: a risk estimate or a forecast exists in several revisions, each tied to an assumption and a date. With no trace of the revision, two apparently identical documents become indistinguishable and the information degrades.

It decays fast: a scenario from six months ago serves as history, not as a current reference. An engine that ranks by textual relevance alone and ignores the date gives answers that are plausible and wrong.

The problem to solve is therefore twofold: to capture knowledge in a form structured enough to query it, and to index the binary material already produced without asking people to rewrite it.

Architecture

The 2009 open-source stack covers these two needs with two distinct families of tools.

For structured capture, a wiki remains the tool that asks the least of whoever writes. MediaWiki 1.14 (February 2009) is a solid engine, and the Semantic MediaWiki extension — version 1.4.2, released in February 2009 — adds what a traditional wiki lacks: you annotate the links between pages with typed properties ([[has risk::credit]], [[revision date::2009-03]]) and query them like a small database. A methodological note stops being free text and becomes a node with navigable attributes. DokuWiki, which stores content in text files rather than a database, remains the lighter alternative when volume is modest and integration with file versioning outweighs everything else.

For indexing material that already exists, the reference is Apache Lucene, a Java full-text indexing library mature for years, and its incarnation as a service: Apache Solr. Solr 1.3, released in September 2008, exposes Lucene as an HTTP service with a configurable schema, faceted search and index replication. Faceted search is what is needed most here: it filters results by attribute — risk type, period, author, asset class — instead of relying on textual ranking alone.

Extracting text from binaries is a separate problem. Apache Tika, young at the time but already usable, reduces PDFs, Office documents and HTML to text plus metadata, and covers precisely the artefacts a financial domain produces in quantity. The chain is this: Tika extracts, Solr indexes with one field for the date and one for the document type, the semantic wiki hosts the knowledge that is born already structured.

Critical point

The crux of the integration lies in none of the components but in the consistency of the schema across them. Solr works well only when the field schema is designed for the domain rather than left generic. A date field declared as a string instead of a date type will not sort chronologically; a risk_type field without a controlled vocabulary generates ten variants of the same value and renders the facet useless. The same constraint applies on the semantic wiki: properties must be defined once and used with discipline, otherwise the structured query returns an arbitrary subset.

The bulk of the work shifts from installing the software to modelling the domain. Deciding which attributes make a document findable in a risk setting — the exposure class, the date of the assumption, the reference scenario — is analysis work, not systems administration. The open-source tool removes the cost of the search infrastructure; the cost of understanding what one is indexing remains.

Implications

The practical consequence is that the schema must be versioned alongside the code. When an attribute is added to the index or a property to the wiki, the whole corpus must be reindexed so that the older documents know about the new field; Solr allows this, but it is an operation to plan, not to improvise. It is therefore worth treating the Solr schema and the set of semantic properties as project artefacts under version control, with a repeatable reindexing procedure.

A second consequence concerns access boundaries. In a financial setting not all knowledge is readable by everyone, and neither Solr 1.3 nor MediaWiki resolves document-level access control natively and granularly. Permission filtering has to be applied upstream — by separating indices or filtering queries by role — and it is an architectural choice best made at the start, because retrofitting it onto an already-populated index is costly.

Limits

The stack described indexes and retrieves; it does not interpret. Solr’s faceted search finds the documents that discuss a given risk, but it does not establish causal relations between scenarios nor reconcile conflicting estimates: that work stays with the analyst. Likewise the semantic wiki makes queryable the knowledge that someone has already structured by hand, and on its own it structures nothing. On the operational side, Solr 1.3 offers no near-real-time indexing: the index updates on explicit commits, with a delay that in a fast-typing workflow must be managed. These are known boundaries of the period, and they mark where what the tool does on its own ends and where the project begins.

The knowledge and activity management platform built on these foundations for Prometeia is described in the insight published by noze: https://www.noze.it/en/insights/knowledge-management-prometeia/.

https://www.mediawiki.org/wiki/Release_notes/1.14 https://semantic-mediawiki.org/wiki/Semantic_MediaWiki_1.4.2 https://solr.apache.org/ https://lucene.apache.org/ https://www.dokuwiki.org/dokuwiki

Cover image: Screenshot of a Semantic MediaWiki page showing structured-data annotations on a wiki article — screenshot by Eloquence, CC BY-SA 3.0 — https://commons.wikimedia.org/wiki/File:Semantic-mediawiki.png