cert-manager treats TLS certificate issuance and renewal as declarative Kubernetes resources: a controller reconciles observed state against desired state, exactly as it happens for any other cluster object. It is developed by Jetstack, written in Go and distributed under the Apache 2.0 licence; it takes over from kube-lego, the Let’s Encrypt tool the same authors have begun to deprecate.

Context

On a cluster with many hosts published through Ingress, managing certificates by hand is repetitive work that comes back at every expiry: you generate the CSR (Certificate Signing Request), complete validation with the CA, store the key and chain in a Secret, then do it all again before the certificate lapses. With Let’s Encrypt the lifetime is 90 days, so renewal comes round every few months for each name. The first generation of automation — kube-lego — read the kubernetes.io/tls-acme: "true" annotation on Ingress resources and obtained ACME certificates, but it nailed all the logic to a single issuer, inside a flow that was hard to extend.

The next step is to describe the certificate as a first-class resource and hand a controller the job of bringing it to the requested state. It is the same model Kubernetes already applies to Deployments and Services, extended to cryptographic material.

Architecture

cert-manager adds a few Custom Resource Definitions under the certmanager.k8s.io group, version v1alpha1. The two main resources are:

  • Issuer (namespace-scoped) and ClusterIssuer (cluster-scoped) — these describe an issuing authority: an ACME account (Let’s Encrypt or another compatible server), an internal CA built from a key/certificate pair in a Secret, or a self-signed issuer for tests.
  • Certificate — this declares the certificate you want: the DNS names (Subject Alternative Names), the destination Secret where key and chain end up, the referenced issuer, and the early-renewal window.

The controller keeps an eye on these resources and acts accordingly: it generates the private key and the CSR, carries validation through with the issuer, writes the result into the named Secret, and schedules renewal ahead of expiry. A ClusterIssuer for Let’s Encrypt is configured like this:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v01.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    http01: {}

Alongside the main controller there is a component of its own, the ingress-shim, which keeps the old annotation-driven flow alive: it intercepts annotated Ingress resources and creates the matching Certificate resource for each, so anyone coming from kube-lego need not rewrite their manifests straight away.

The ACME flow and the critical point

For ACME issuers the controller has to prove to the CA that it controls the domain before it gets the certificate. The Automatic Certificate Management Environment protocol offers several challenge types, and choosing between them is where automation lays hands on the network and the DNS: it should be decided with care.

  • HTTP-01 — the CA asks for a token at http://<domain>/.well-known/acme-challenge/<token>. cert-manager publishes that path for a moment through the Ingress. It works only when the name is reachable from the outside on port 80, and it does not cover wildcard certificates.
  • DNS-01 — the proof is a TXT record at _acme-challenge.<domain>, created through the DNS provider’s API and removed at the end. It does not require the service to be exposed and it enables wildcards, but in return cert-manager receives credentials with write access to the DNS zone.

A third, historical challenge — TLS-SNI-01 — is by now to be treated as out of use: on 9 January 2018 Let’s Encrypt suspended it after a report that, on certain shared-hosting configurations, it could be used to obtain certificates for subdomains the requester did not control. It is a reminder of a structural fact: to the CA, whoever can answer the challenge is the rightful holder of the name. Domain validation attests control of the endpoint, not the identity of whoever stands behind it.

ACME v2 and wildcards: where things stand in February 2018

The ACME protocol is in transition. On 5 January 2018 Let’s Encrypt published the ACME v2 staging endpoint, at https://acme-staging-v02.api.letsencrypt.org/directory, and gave 27 February as the planned date for the production endpoint. ACME v2 is not backwards compatible with v1 and is the version that will bring wildcard certificate issuance, allowed only with the DNS-01 challenge.

For now, then, production issuance still goes through ACME v1 (acme-v01.api.letsencrypt.org), wildcards via Let’s Encrypt are not yet in production, and full v2 support in cert-manager depends on the installed version, to be checked case by case. Anyone with wildcards on Kubernetes in mind would do well to set up a DNS-01 issuer with a supported provider straight away, to be ready once v2 production goes live.

Operational implications

Bringing certificates inside the cluster API shifts the risks and the controls. Private keys end up in Secret objects, which by default are base64-encoded but not encrypted at rest in etcd: enabling encryption at rest and a restrictive RBAC (Role-Based Access Control) policy on Secret access becomes part of the surface to protect. For DNS-01 issuers the DNS provider credentials live in the cluster too, and must be treated as high-privilege secrets: whoever holds them can issue certificates for the whole zone.

A single ACME issuer also tends to concentrate requests: Let’s Encrypt applies rate limits per registered domain, and a misconfigured controller that retries in a loop can burn through the quota and block legitimate issuance. The staging endpoint exists precisely to avoid this while you tune the setup.

Limits

cert-manager is still at an early stage: the CRDs are v1alpha1, so the shape of the resources can change between versions without backwards compatibility, and manifest migration has to be budgeted for. The controller automates issuance, but it does not remove the need to monitor: you still have to follow events on Certificate resources and the actual expiry of the certificates in their Secret objects, because a failed renewal makes no noise until the certificate lapses. And domain validation remains domain validation: where assurance about organisation identity is needed, the options to weigh remain an internal CA via Vault or an enterprise issuer, not ACME automation.

Cover image: ACME (Automatic Certificate Management Environment) protocol icon/logo — logo by IETF ACME Working Group, public domain — https://commons.wikimedia.org/wiki/File:ACME%E2%80%93protocol-icon.svg