Developer/Observability/Metrics/Fundamentals/metric-types-counters-gauges-histograms

Metric Types: Counters, Gauges, and Histograms

Concept

Not every measurement behaves the same way over time, and picking the wrong metric type for a value silently breaks the math that dashboards and alerting rules do on top of it later. The three core types (as codified by Prometheus, the de facto standard vocabulary most observability tooling now shares) map to three fundamentally different shapes of data:

Counter — a value that only ever increases (or resets to zero on process restart), used for cumulative totals: requests served, errors raised, bytes sent. A counter is never read directly; it's read as a rate of change over a window (rate(http_requests_total[5m])), because the raw cumulative number ("14,802,331 requests since the process started") is meaningless on its own.
Gauge — a value that can go up or down arbitrarily and is meaningful to read at a single point in time: current memory usage, number of open connections, queue depth, temperature. Unlike a counter, a gauge's instantaneous value is the useful number — no rate calculation needed.
Histogram — buckets individual observations (typically durations or sizes) into configurable ranges and tracks a count and sum for each bucket, which lets you approximate percentiles after the fact (histogram_quantile(0.99, ...)) from data already collected, without deciding up front exactly which percentile you'll need.

A closely related fourth type, the Summary, calculates quantiles client-side at observation time instead of via buckets — cheaper to query but the computed quantiles can't be aggregated correctly across multiple instances (you cannot average two p99s and get a valid p99), which is why histograms are generally preferred for anything that needs to be aggregated across a fleet of replicas.

Tradeoffs

Type	What it answers	Aggregatable across instances?	Common mistake
Counter	"How many total / at what rate?"	Yes — sum the rate across instances	Reading the raw cumulative value instead of a `rate()`/`increase()` over a window
Gauge	"What's the value right now?"	Yes for sums/averages of current state (e.g. total memory across a fleet)	Using a gauge for something monotonic (should be a counter) — loses the ability to compute a clean rate
Histogram	"What's the distribution (median, p95, p99)?"	Yes — bucket counts sum correctly across instances before computing the quantile	Choosing bucket boundaries that don't match the actual data range, making the resulting percentile estimate coarse or meaningless
Summary	"What's the distribution, cheaply, on one instance?"	No — client-computed quantiles cannot be validly combined	Treating a summary's quantile as fleet-wide when it was only ever computed against one instance's traffic

The real tradeoff for histograms specifically is bucket selection: too few or badly-placed buckets (e.g. buckets at 100ms/500ms/1s when real latency clusters between 10-50ms) makes the percentile estimate useless even though the metric type itself is correct — the shape of the buckets has to be tuned to the actual data, not left at a library default.

When to use / when not to

Use a counter for anything that only accumulates — total requests, total errors, total bytes processed. If a value can go down, it's not a counter.
Use a gauge for anything read as a current snapshot — in-flight requests, queue length, active database connections, memory/CPU usage.
Use a histogram the moment you care about the distribution of a value, not just its average — request latency and payload size are the two most common cases, since an average latency can look fine while p99 users are having a terrible time.
Avoid defaulting to average/mean for latency dashboards — a mean hides exactly the tail-latency problem (a small fraction of very slow requests) that a histogram-derived p95/p99 is built to surface.
Prefer a histogram over a summary whenever the metric needs to be aggregated across multiple service replicas (the normal case in any horizontally-scaled service) — a summary's per-instance quantiles simply cannot be combined correctly after the fact.

Common pitfall

Modeling a monotonically-increasing-but-occasionally-reset value (like a request counter that resets to zero on every deploy) as a gauge instead of a counter, which breaks rate()-style calculations: a counter's reset-to-zero is a known, handled case in query languages built around counters, while a gauge dropping to zero looks like a real drop in whatever the gauge measures and produces a misleading dashboard spike or alert.

Engineering Lens

Metric-type selection is a cheap decision made once per instrumented value, but it's the decision that determines whether every downstream dashboard and alert built on that metric is even mathematically sound — a p99 latency alert built on an average, or a rate-of-change alert built on a gauge, will eventually fire (or fail to fire) on data that never meant what the query assumed it meant. In a review, the strong answer isn't "we have metrics on this" — it's being able to say why a given signal is a counter vs. a gauge vs. a histogram and what that choice specifically enables (a valid fleet-wide rate, a valid fleet-wide percentile) that the wrong type would have silently broken. This transfers identically across domains: a trading system's order-latency histogram and a checkout service's payment-latency histogram are solving the exact same aggregation problem.