Hermes Wiki

Mutual TLS (mTLS)

Concept

Ordinary TLS is one-directional trust: a client connecting to api.example.com verifies the server's certificate against a trusted certificate authority, gets encryption and server authenticity, and that's it. The server has no cryptographic idea who's calling — it might check an API key or a bearer token at the application layer, but at the transport layer any client can open the connection. That asymmetry is fine for a browser talking to a public website. It's a gap in a service-to-service environment, where "which service is this request actually coming from" is a security-relevant question that a shared network alone can't answer — a compromised pod on the same cluster network can otherwise call any internal endpoint it can reach.

Mutual TLS closes that gap by making authentication bidirectional. During the TLS handshake, the client presents its own certificate alongside the server's, and each side validates the other's certificate against a trusted CA before the connection completes. Both ends now cryptographically know who they're talking to before a single byte of application data moves — not "some client with a valid token," but "this exact identity, backed by a certificate issued by an authority we trust." The practical effect is that identity and encryption become one mechanism instead of two: there's no separate credential (API key, static token) to leak, rotate, or accidentally log, because the identity check is the TLS handshake.

The operational cost is a certificate lifecycle that now has to run continuously and automatically — every workload needs a short-lived certificate, issued by an internal CA, renewed before expiry, and revoked when a workload is decommissioned or compromised. Doing this by hand across more than a handful of services isn't viable, which is exactly why mTLS in practice is almost always delivered by infrastructure rather than hand-rolled per service: a service mesh's control plane acting as the CA and rotating sidecar certificates automatically, or a platform like SPIFFE/SPIRE issuing workload identities independent of any specific mesh product.

Tradeoffs

Approach Client identity at transport layer Credential leak surface Operational cost Failure mode
One-way TLS + API key/token at app layer None cryptographic — token is a bearer secret High — token can leak via logs, config files, source control Low to set up A leaked token grants access from anywhere; no transport-layer signal that the caller isn't who it claims
mTLS, manually managed certs Strong — certificate-backed identity Low leak surface, but manual rotation invites long-lived or expired certs sitting unrotated High — someone has to track expiry and rotate by hand per service Silent outages when a forgotten cert expires; delayed revocation if rotation is infrequent
mTLS, automated via mesh/SPIFFE Strong Very low — certs are short-lived and auto-rotated, nothing long-lived to leak High to build and run the CA/control-plane infra, low per-service marginal cost after that Control plane itself becomes a critical dependency — its outage can block new connections cluster-wide

The real trade isn't "encryption vs. no encryption" — one-way TLS already gives you that. It's whether client identity is a cryptographic fact enforced at the network layer, or an application-layer assertion backed by a credential that can be copied, replayed, or forgotten in a log file. mTLS buys the former, at the cost of a certificate lifecycle that has to be automated to be sustainable at any real scale — automating it is where most of the actual engineering effort goes.

When to use / when not to

  • Use for service-to-service traffic inside a trust boundary you're trying to make zero-trust (see Zero Trust Architecture) — internal APIs, microservice-to-microservice calls, anywhere "which service is this" needs to be a verifiable fact rather than an assumption based on network location.
  • Use when regulatory or compliance requirements call for strong, auditable service identity — payments and financial-services environments frequently require exactly this.
  • Use when you already have (or are adopting) a service mesh or a workload-identity platform like SPIFFE/SPIRE — the marginal cost of turning on mTLS is low once that infrastructure exists.
  • Skip it, or defer it, if you have no automated certificate lifecycle and no near-term plan to build one — manually managed mTLS certs are a reliability liability (silent expiry outages) that can be worse than the security gap it closes.
  • Skip it for public-facing client-to-server traffic where the "client" is an arbitrary browser or mobile app — there's no fixed client identity to issue a certificate to; that's the traditional one-way-TLS-plus-application-auth case, not an mTLS use case.

Common pitfall

Standing up mTLS without solving certificate rotation first. It works perfectly in a demo — certs get issued once, everything connects — and then fails months later when the first batch of certificates expires simultaneously, because nobody built the automated renewal path before turning mTLS on. The result is a cluster-wide outage caused by the exact security control that was supposed to make things safer, and it's why mTLS is treated as infrastructure to adopt (via a mesh or a workload-identity platform) rather than a feature to bolt on service-by-service.

Engineering Lens

mTLS is a useful test case for a broader Principal-level judgment: does this control need to be a platform capability or can it stay a per-team choice? Client identity at the transport layer is exactly the kind of thing that's cheap to get right once, centrally, and expensive and inconsistent to get right N times, once per team — the same shape of argument that justifies a service mesh or a centralized secrets platform. In an architecture review, the sharper question isn't "should we use mTLS" but "who owns the CA and the rotation SLA, and what happens to live traffic if that piece goes down" — because a control-plane outage on a mesh-delivered mTLS setup can now block legitimate service-to-service calls, trading one failure mode (unauthenticated traffic) for another (a new critical dependency). That tradeoff, argued explicitly rather than assumed away, is what separates "we turned on mTLS" from a defensible security architecture.

Sources

Hermes Wiki