On 17 February 2015 the Graylog project published version 1.0, the first marked stable after years of development under the graylog2 name. The platform gathers logs from heterogeneous hosts into a single search point, and keeps the index separate from configuration metadata, across two distinct engines.

Context and problem

In a mid-sized estate logs appear everywhere: sshd and sudo on the servers, web servers, firewalls, switches, applications with their own formats. As long as they sit in /var/log on each machine, a cross-cutting search — “who authenticated as root from this IP in the last hour” — means running ssh onto every host and grep by hand. For a security investigation it is not practical, and the attacker who wipes the local logs wipes the only copy that exists.

Centralising logs answers two different needs: keeping a copy of events out of reach of a compromised host, and querying the whole set with one query. The classic syslog stack (rsyslog, syslog-ng) covers transport and forwarding, but stops at remote text files. To search in a structured way you need an index.

In 2015 the most common answer is the ELK stack — Elasticsearch, Logstash, Kibana. Graylog covers the same ground but splits the work differently: ingestion, routing, alerting and a search interface sit in a single application server, and Elasticsearch is left the sole role of indexing backend.

Architecture

Graylog 1.0 runs on three processes:

  • graylog-server — a JVM (Java) application that receives messages, processes them and serves the REST API
  • Elasticsearch — storage and full-text search of indexed messages
  • MongoDB — configuration metadata: stream definitions, dashboards, users, alarms

Keeping the two datastores separate is the central architectural decision. The logs go on Elasticsearch — high-volume, append-only data with time-based retention; the configuration sits on MongoDB — low-volume, but it must survive rebuilding an index. If you drop and reindex Elasticsearch you do not lose streams and dashboards, because they live elsewhere.

The web interface (in 1.0 a single-page application served by the server itself, after the old Ruby graylog2-web-interface was merged in) talks to the server only over the REST API. That same API is the attachment point for automation and external integration.

Ingestion and GELF

Ingestion goes through inputs: listeners configured by protocol and port. Graylog 1.0 receives syslog over UDP and TCP, GELF over UDP/TCP/HTTP, and raw TCP/UDP for unstructured text.

syslog as a transport has a known and by now dated limitation: RFC 3164 (the 2001 “BSD syslog”, in fact an after-the-fact description of practice that already existed) guarantees no minimum message length and packs priority, timestamp and text into a single line to be decoded with some heuristics. RFC 5424 from 2009 brings structured data and normalised timestamps, but in the field adoption stays partial.

GELF — Graylog Extended Log Format — exists to get around those limits. A GELF message is a JSON object with fixed fields (version, host, short_message, timestamp, level) and free additional fields, prefixed with an underscore (_user_id, _request_path). Two details bear on anyone shipping logs from their own applications:

  • chunking — over UDP a message can be split across several datagrams, each with a magic-byte header, message ID and sequence number for reassembly. The limit is 128 chunks per message and full arrival within 5 seconds, beyond which partial fragments are thrown away
  • compression — over UDP the payload can be GZIP or ZLIB, recognised by the magic byte; over TCP, instead, GELF has neither chunking nor compression and expects one null-terminated payload per line

The concrete advantage of GELF is that the structured field is already there at source: the application emits _order_id=42 as a field, not as a string to be cut out further downstream with a brittle regex.

The delicate point: extraction and extractor order

When a log arrives as unstructured syslog text, the useful information has to be pulled out after ingestion. Graylog does this with extractors: rules applied to an input’s messages that derive fields from a portion of text — with regexes, grok patterns, substring parsing or splitting on a separator. A grok extractor on an Apache access log produces clientip, verb, response, bytes as queryable fields.

The operational detail you pay for at the first serious deployment is order. An input’s extractors run in sequence, and one can depend on a field produced by the one before. Getting the order wrong means the downstream rule cannot find the field it expects and fails silently: the message still enters the index, but without the expected fields, and the query that looks for them simply does not return it. No visible error: an event that exists and that search cannot see.

The same caution applies to extraction conditions. An extractor that runs on every message of a high-volume input, with an unanchored regex, costs CPU on every log. Under load this shows up as a processing backlog — the journal of messages awaiting indexing grows, and logs appear in search minutes late. For a live security investigation, the lag between event and visibility is already a security property in itself.

Streams, alarms and the SIEM context

Streams route messages into logical flows with field rules: every message with level ≤ 3, or everything from a given source. A stream is also the unit on which alarms are defined — conditions such as “more than N messages in M minutes” or “presence of a field” — with notification by email or HTTP.

This combination of inputs, streams and alarms covers part of what is asked of a SIEM (Security Information and Event Management): centralised collection, normalisation, search and threshold alerting. True correlation across events from different sources — rules crossing a failed login with a successful access right after — remains to be built by hand on top of streams and alarms, and here the boundary with commercial SIEM products is still sharp.

Limits

Graylog 1.0 carries the coupling to the Elasticsearch version of the period: the alignment between the two versions is not free, and a backend upgrade must be checked against server compatibility. The 1.0 processing pipeline rests on per-input extractors, without yet an input-independent rule language: a complex transformation means chaining extractors or pre-processing upstream.

On licensing, the server code is GPLv3 and the GELF format is documented and implementable by third parties; GELF libraries for the common languages exist, maintained outside the main project, with uneven quality and coverage.

Cover image: Graylog text logo: the lowercase wordmark “graylog”, with “gray” lighter and “log” darker — logo by Graylog, public domain — https://commons.wikimedia.org/wiki/File:Graylog_logo.svg