LocalzDocs/Observability-Stack

Observability Stack

The 1-minute mental model

There are only 4 observability questions in any system:

Is the system alive? → Metrics
Why did it break? → Logs
Where did it break? → Traces
How much money did we make/save? → Business analytics

Every tool answers only one of these well. If two tools answer the same question → overkill.

Tool → Problem Mapping

1) Metrics (Is it healthy?)

Primary purpose: CPU, memory, latency, error rate, SLO/SLA, "Are we on fire?"

Best tool: Prometheus

Pull-based (safe), Cheap, Kubernetes-native, Numeric time series compress extremely well
Add Thanos / Mimir / VictoriaMetrics only when scale explodes

For Localz: ✅ Prometheus = mandatory | ❌ Datadog metrics = optional luxury

2) Logs (Why did it break?)

Primary purpose: Stack traces, Error context, Debugging prod issues

Best tool: Elasticsearch / OpenSearch

Free-text search, Correlation, Time-based queries

Critical rule: Elasticsearch is for recent logs, not history. (Hot: 7–14 days, Warm: 30–90 days, then delete or archive)

For Localz: ✅ OpenSearch = yes, with short retention + S3 offload | ❌ Multi-year ES retention = never

3) Traces (Where exactly did it break?)

Primary purpose: Distributed request flow, Latency attribution, "Which service caused this?"

Enterprise tools: Dynatrace, Datadog APM, New Relic

Open alternative: OpenTelemetry + Tempo/Jaeger

For Localz: ❌ Dynatrace = overkill | ✅ OpenTelemetry (later, optional)

4) Errors (What users are hitting?)

Primary purpose: Application exceptions, Stack traces tied to user flows, Release regression detection

Best tool: Sentry

Dev-focused, Extremely high signal, Cheap vs APMs

For Localz: ✅ Sentry = high ROI | ❌ Don't replace with logs alone

5) Business analytics (How much money / value?)

Primary purpose: MRR, GMV, Cost saved, Cohort analysis — this is NOT observability, it's decision intelligence

Enterprise tools: Snowflake, Amazon Redshift, BigQuery, ClickHouse

Why NOT Elasticsearch: ES is expensive for scans, poor for joins, not a system of record

For Localz: ❌ Snowflake now = too early | ✅ Postgres + batch aggregates | ➕ Warehouse later (when revenue exists)

Why enterprises look "tool-heavy"

Because one tool cannot violate physics:

Problem	Data shape	Best tool
Metrics	numeric, append-only	Prometheus
Logs	text, high-cardinality	Elasticsearch
Traces	graphs, spans	APM
Errors	stack traces	Sentry
Business KPIs	joins, scans	Warehouse

Trying to force one tool to do all → cost + pain.

The minimal sane stack for Localz

Phase 1–2 (start here)

Tool	Keep?	Why
Prometheus	✅	Core health
OpenSearch	✅	Logs (short)
Sentry	✅	App quality
Postgres	✅	Truth + KPIs
S3	✅	Log archive

Explicitly skip

Datadog (too expensive)
Dynatrace (enterprise APM)
Snowflake (premature)
Redshift (premature)

When to add "enterprise toys"

Tool	Add when
Warehouse	>$20–30k MRR
APM	Latency debugging hurts
Datadog	You hire SREs
Dynatrace	Regulated enterprise customer demands it

One killer rule

Observability tools should reduce thinking, not add thinking.

If a tool doesn't prevent incidents, shorten outages, or improve release quality → it doesn't belong yet.

Clean Localz observability architecture

                     ┌─────────────────────────────────────────────┐
                     │                 Localz Apps                 │
                     │  FastAPI services, workers, cronjobs, etc.  │
                     └───────────────┬───────────────┬─────────────┘
                                     │               │
                (A) Metrics          │               │ (B) Logs
      /metrics endpoint + lib        │               │ stdout/stderr JSON
                                     │               │
                                     ▼               ▼
                           ┌────────────────────────────────┐
                           │     Kubernetes / EKS Cluster    │
                           │  DaemonSets / Sidecars / Agents │
                           └───────┬───────────────┬────────┘
                                   │               │
                 (C) Traces        │               │ (D) Errors
            OpenTelemetry SDK      │               │ Sentry SDK
                                   │               │
                                   ▼               ▼
                      ┌──────────────────┐     ┌───────────────────┐
                      │  OTel Collector  │     │      Sentry        │
                      │  (central router)│     │ (errors, releases) │
                      └───┬─────────┬────┘     └───────────────────┘
                          │         │
               traces →   │         │  optional: metrics/logs routing too
                          ▼         ▼
                  ┌────────────┐   ┌────────────────────────┐
                  │   APM      │   │   Prometheus (+Grafana) │
                  │ (Tempo/    │   │   metrics scraping      │
                  │  Jaeger)   │   └────────────────────────┘
                  └────────────┘

(B) Logs pipeline (separate, high-volume)
     Pods stdout → Fluent Bit / Vector → OpenSearch (HOT 7–30d)
                                     └→ S3 (archive 90–365d)
                                         └→ Glacier (years, optional)

(E) Business KPIs (separate, "analytics")
     App emits business events (OrderPlaced, SubscriptionRenewed, etc.)
              → Kafka topics
              → stream/batch ETL jobs
              → Warehouse (Snowflake/Redshift/etc.)
              → BI dashboards

Routing rules (what goes where)

1) Metrics → Prometheus

What: Request rate, latency, error rate (RED metrics), CPU/memory, DB connection pool, queue depth
How: Your service exposes /metrics → Prometheus scrapes it → Grafana reads Prometheus for dashboards
Why Prometheus: Metrics are numeric time series → compress well → cheap → fast alerting

2) Logs → Elasticsearch / OpenSearch

What: Structured JSON logs to stdout/stderr, Stack traces, Audit-ish operational logs
How: Pods write to stdout → Fluent Bit / Vector (daemonset) → ES/OpenSearch (hot) → S3/Glacier (archive)
Key point: ES/OpenSearch is a hot search index, not your "forever storage"

3) Errors → Sentry

What: Unhandled exceptions, Performance traces inside one request, Release versions
How: Sentry SDK inside your apps → sends events directly to Sentry → issue grouping, regression detection
Why separate from logs: Sentry gives high-signal "what broke" views, not raw text search

4) Traces → OpenTelemetry → APM backend

What: Distributed traces, request → services → DB → external calls, Spans with timings and attributes
How: OTel SDK in apps → OTel Collector → Tempo/Jaeger (self-host) or vendor APM
Why not "Prometheus for traces": Traces are a graph of spans; metrics are numbers. Different shape.

5) Business KPIs → Warehouse (analytics)

What: OrderPlaced, PaymentCaptured, SubscriptionRenewed, SellerActivated events
How: App emits events to Kafka → ETL → Warehouse → BI dashboards
Why not Elasticsearch: KPIs need joins, long history, cohorts, finance-grade correctness

Why OTel Collector is the "router"

Think of OTel Collector as the reverse proxy for observability: your apps export to one internal endpoint → Collector fans out to the right backend(s).

So your apps don't need to know 10 vendors. They speak one protocol.

Minimal Localz implementation plan

Week 1 (must-have): Metrics: Prometheus + Grafana | Logs: Fluent Bit → OpenSearch (7–14 days) | Errors: Sentry SDK

Week 2–3 (high ROI): Traces: OpenTelemetry SDK + OTel Collector + Tempo/Jaeger

Later (when business grows): Business KPIs to warehouse; keep dashboard-friendly aggregates in Postgres/OpenSearch

One concrete mapping example (same incident across tools)

"Checkout is slow."

Prometheus: latency p95 spiked
Traces: slowdown is in payments-service -> db query
Logs: show DB timeout / retry storm
Sentry: shows new exception after latest deploy
Warehouse: shows conversion dropped today (business impact)

Each tool answers a different question — together they give the full story.