The SMARTOUR compute infrastructure hosts software produced by different partners on release cycles unrelated to one another: we chose to isolate each partner in a dedicated space and to declare the whole platform as code. The constraint was there from the start, and it shaped every later decision about the stack.

Context

SMARTOUR is a research project in which several organisations deliver software that has to coexist on one infrastructure. One partner builds a geolocation service, another a territorial-data API, a third a frontend for public administration. None of them knows the runtime requirements of the others in advance, and the delivery dates do not coincide.

A traditional shared host, with services running on the same operating system, would have forced continuous coordination over library versions, language runtimes and system dependencies. Containerisation removes that coordination at the root: each deliverable arrives with its dependencies packaged, and the host has only to run the image. The container format follows the Open Container Initiative specifications — image-spec and runtime-spec, at version 1.0 since 2017 — so the project is not tied to one runtime’s proprietary format.

Architecture

The orchestration layer is Kubernetes, at 1.18, out in March 2020 and described by the maintainers as a fit-and-finish release rather than one of new features. We did not choose it for novelty: Kubernetes offers a declarative model in which the desired state of the system is a set of versionable YAML objects, and a control loop continuously reconciles the actual state towards the declared one. For an infrastructure that has to be rebuildable by third parties at the end of the project, that property weighs more than any single feature.

On top of Kubernetes we put Rancher 2.4 for multi-cluster management. Rancher 2.4 manages thousands of clusters from a single control plane and introduces CIS Scan, which checks RKE clusters against the Center for Internet Security benchmarks. SMARTOUR does not need thousands of clusters, but it does need a single point from which to assign namespaces, govern roles through role-based access control (RBAC), and give each partner a view restricted to its own space without granting it the whole cluster.

Per-partner isolation rests on two native Kubernetes primitives:

  • the Namespace, which logically separates resources and acts as a scope for names, quotas and policies;
  • the NetworkPolicy, which limits traffic between pods on label selectors, so one partner’s service does not reach another’s without an explicit permission.

To these we add ResourceQuota and LimitRange, to stop a misconfigured deployment from eating the others’ resources. This is isolation at the software level, not the kernel: containers share the node’s kernel, and I return to that limit below.

The pipeline is GitLab CE. Version 13.0, out on 22 May 2020, brings the integrated container registry and Auto DevOps. We want to keep three things in one self-hosted tool: a Git repository, a private image registry and CI/CD. A partner pushes code, the pipeline builds the image, publishes it to the internal registry, and the Kubernetes deployment reads from there. No image passes through external services, and this simplifies the trust model on infrastructure that hosts third-party code.

For packaging deployments we use Helm 3, out in November 2019. Version 3 removes Tiller, the server-side component that in Helm 2 ran inside the cluster with broad privileges: Helm now renders charts client-side and relies directly on the RBAC of the kubeconfig. On a cluster shared between partners, removing a component with cluster-wide permissions cuts the attack surface in a concrete way.

Observability is Grafana, with metrics scraped by Prometheus. For centralised logs we chose Loki, the Grafana Labs aggregation system that indexes only a set of labels per stream instead of the full content: on a multi-partner cluster this keeps log storage cost low and still lets us filter by namespace, and so by partner.

The critical point

The part that needed most attention is the security boundary between partners, more than the orchestration itself. Namespaces and NetworkPolicies isolate logical resources and network traffic, but every partner’s containers share the same node kernel. A container-escape vulnerability — the class of bug by which a process inside the container reaches the host — makes the whole isolation permeable.

The mitigation is defensive and layered: PodSecurityPolicy to forbid privileged containers and hostPath use, images built on minimal bases to shrink the surface, and workload separation across nodes with taints and tolerations where a partner has stricter isolation requirements. None of these measures removes the shared kernel; they lower the probability of exploitation. Kernel-level isolation would call for sandboxes such as gVisor or lightweight virtual machines such as Kata Containers: we assessed them and set them aside, so as not to pile more operational complexity on a small team.

Implications

The stack is all open source and self-hosted, and this is a project requirement rather than a preference. A funded research deliverable has to be rebuildable and verifiable by third parties after handover: any component under a proprietary licence or dependent on a closed cloud service would be an obstacle to reproducibility. Kubernetes, Rancher, GitLab CE, Helm, Prometheus, Grafana and Loki carry licences that allow reuse and redistribution, and the hardware is on-premises.

The operational consequence is that responsibility for security and updates rests entirely with the team, without the cover of a managed provider. Kubernetes updates follow a cadence of three minor releases a year, each with a limited support horizon, and this imposes a maintenance plan to be sized from the start.

Limits

The current state is a prototype. Per-namespace isolation is enough for partners collaborating inside the same project, but it does not equal a multi-tenant boundary between parties that do not trust each other. The Kubernetes control plane stays a single point of failure until it is replicated, and at this stage it does not run in high availability. Log collection with Loki is recent and its operational maturity is still to be confirmed under real load. These are choices consistent with a research deliverable, not with a continuously running production service, and the distinction is worth keeping explicit.

The evolution of this platform towards a cloud-native architecture is traced in a noze insight on SMARTOUR’s path to Kubernetes: https://www.noze.it/en/insights/smartour-kubernetes/.

https://kubernetes.io/blog/2020/03/25/kubernetes-1-18-release-announcement/ https://www.infoq.com/news/2020/04/rancher-kubernetes-clusters/ https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/ https://helm.sh/blog/helm-3-released/ https://github.com/opencontainers/image-spec

Cover image: Kubernetes logo: a stylised seven-spoke ship’s helm wheel in white on a blue hexagonal background — logo by Google, Inc., CC BY 4.0 — https://commons.wikimedia.org/wiki/File:Kubernetes_logo_without_workmark.svg