Grid middleware solves two problems that standard web protocols leave open: how to keep the state of a remote resource between one call and the next to services that are stateless by design, and how to authorise a process to act on behalf of a user who, while that process runs, is not at the keyboard. Globus Toolkit 4 (GT4) tackles both by leaning on public specifications — WSRF for the first, X.509 proxy certificates for the second — instead of inventing protocols of its own. Looking at how it does so is worth the effort: GT4’s choices are a case study in what it means to build distributed infrastructure by reusing the standards that already exist.

Context

Globus Toolkit comes out of the Globus Alliance, with Ian Foster (Argonne National Laboratory, University of Chicago) and Carl Kesselman (USC Information Sciences Institute) as the principal figures. Version 4.0 was released in May 2005 under the Apache 2.0 licence; as I write, the 4.0.x line is the reference. Production infrastructure runs on GT4: TeraGrid in the United States, the Open Science Grid, and the European EGEE project (Enabling Grids for E-sciencE), whose gLite 3.0 middleware — released in May 2006 — includes Globus components through the Virtual Data Toolkit. The Worldwide LHC Computing Grid, built at CERN to distribute analysis of Large Hadron Collider data across a Tier-0/Tier-1/Tier-2 hierarchy of sites, sits on the same layer.

The architectural transition that produced GT4 came earlier, and reconstructing it explains today’s choices. Version 3 implemented OGSI (Open Grid Services Infrastructure), published by the Global Grid Forum in 2003. OGSI packed grid concepts into web services in one monolithic specification, and it fit badly with the SOAP and XML tools of the day: adopters often ended up patching or replacing the standard toolkits. In January 2004 IBM and the Globus Alliance broke those ideas down into WSRF (Web Services Resource Framework), a family of separate specifications, later moved under OASIS. GT4 is the first Globus release built entirely on WSRF.

Architecture: giving state to a stateless service

A SOAP web service, by design, keeps nothing between one call and the next. A job running on a remote cluster, by contrast, has state that changes over time — queued, running, finished — and the client must be able to query and modify it. WSRF resolves this tension by separating the service (stateless, addressable) from the resource (stateful, holding its own data). The pairing of service plus resource identifier is called a WS-Resource, addressed through a WS-Addressing endpoint reference.

A resource’s properties are exposed as WS-ResourceProperties: XML documents queried and modified through standard operations, so each service need not reinvent its own API for reading state. Lifetime is governed by WS-ResourceLifetime, which provides both explicit destruction and a scheduled expiry — the resource self-destructs if it is not renewed, and this stops orphaned state from piling up on a remote site. Change notifications travel over WS-BaseNotification. In practice, the four historical families of Globus components — execution management (WS-GRAM), data movement, security, monitoring (MDS) — are reimplemented as WS-Resources and become interoperable with any conformant web-services stack.

The clearest case is WS-GRAM (Grid Resource Allocation and Management). It takes a job description, represents it as a queryable WS-Resource, and translates it into commands for the site’s local scheduler — PBS, Condor, LSF, SGE. The client always queries the same abstract interface; the underlying scheduler stays hidden. Movement of the job’s input and output files is handled by GridFTP, an FTP extension that opens multiple parallel TCP streams to saturate high-latency wide-area networks and resumes from the break point when a transfer is interrupted — indispensable once volumes reach terabytes.

The critical point: delegating credentials

The subtler problem is not authenticating a user but authorising a process that acts while the user is elsewhere. A researcher submits a workflow that, over hours, launches jobs on one site, moves the results to a second, and starts further jobs on a third. Each step needs their identity, yet they are not at the keyboard typing a passphrase. The Grid Security Infrastructure (GSI) handles this with proxy certificates, standardised as RFC 3820 in 2004.

A proxy certificate is a short-lived X.509 certificate, signed by the user’s identity certificate (or by a preceding proxy, in a chain) and carrying an unencrypted private key. It is the clear key that lets automated processes sign without human intervention; it is the short lifetime — usually twelve hours — that bounds the damage if the proxy is exfiltrated. Delegation is the mechanism by which site A, having received the proxy, mints a further one to act towards site B on behalf of the same user, without ever transmitting the original identity key. Trust between institutions rests on an International Grid Trust Federation of certification authorities that recognise one another.

The point not to miss is that GSI does not stand up a security system from scratch: it composes X.509, TLS, and an extension standardised at the IETF. WSRF does the same relative to SOAP and WS-Addressing. This is the difference between extending standards and replacing them, and GT3 with OGSI had taken the second road and paid for it in interoperability.

Implications

For anyone designing distributed systems, the value of GT4 as a reference lies in the decisions rather than the code. Federating heterogeneous resources while leaving each site its administrative autonomy — not one supercomputer, but independent institutions sharing capacity under agreed policy — is the same problem that returns across many multi-organisation architectures. The split between stateless service and stateful resource, authorisation by delegation to short-lived credentials, and the translation of one abstract interface onto heterogeneous backends are patterns that hold well beyond scientific computing.

Limits

WSRF has created friction within the web-services community. Its overlap with WS-Transfer and WS-Management, backed by another group of vendors, has fragmented the specification space for managing stateful resources, and it is not given that WSRF convergence will prevail. The operational complexity of installing and configuring GT4 remains high: adoption almost always goes through intermediate distributions (gLite, VDT) that package it. Finally, GSI presupposes a working, mutually recognised PKI across institutions — an organisational cost, before a technical one, which scientific grids carry thanks to trust federations that already exist and which, outside that context, cannot be taken as given.

Cover image: Close-up top view of a server rack packed with colored network cables in the CERN data centre — photo by Marián Hubinský, CC BY-SA 4.0 — https://commons.wikimedia.org/wiki/File:Rack_of_Worldwide_LHC_Computing_Grid.jpg