sota-observability

SOTA Observability & Reliability

Purpose

Make every production system answerable. Two questions define success:

1. "Why is this request slow/failing?" — answerable for any single request from a trace ID, without adding new instrumentation.
2. "What broke at 3am?" — answerable from symptom-based alerts that page only when users are hurt, each linked to a runbook and a dashboard that narrows cause in minutes.

This skill covers structured logging, metrics, distributed tracing, SLOs and alerting, and operational readiness — both how to build them correctly and how to audit them adversarially. Telemetry is a product with users (on-call engineers) and costs (storage, cardinality, attention). Treat both.

BUILD mode

When writing or modifying code, apply the rules files as design constraints, not afterthoughts. Workflow:

1. Identify the signal need before coding. For each new endpoint, job, or consumer: which SLI does it affect, what one wide event describes a unit of work, what spans bound its external calls.
2. Instrument with OpenTelemetry API (not vendor SDKs) in libraries; configure SDK/exporters only at the application entry point. Follow OTel semantic conventions for names and attributes.
3. Emit one canonical wide event per request/job at completion, carrying trace_id, outcome, durations, and business context. Debug logs are supplementary, sampled, and disposable.
4. Propagate context everywhere: W3C traceparent over HTTP, injected into queue message headers, restored in consumers and scheduled jobs.
5. Redact at the logger, never at call sites. Denylist+allowlist serializers for PII/secrets; fail closed on unknown object dumps.
6. Budget cardinality. Every metric label must have a known, bounded value set. No IDs, no URLs, no user input in labels.
7. Ship the operational surface with the feature: health endpoints with correct liveness/readiness semantics, dashboard panels answering the questions the feature raises, burn-rate alerts wired to the SLO, runbook entry for each new alert.
8. Verify by simulation: kill a dependency, send a slow request, trigger an error — confirm the trace, the wide event, the metric, and the alert all show it, and that they cross-link (exemplars, trace_id in logs).

AUDIT mode

Assess an existing codebase/deployment. Read rules/06 first for the full playbook; sample real code paths, do not trust README claims.

Severity conventions:

Finding format (one per finding):

[SEVERITY] <short title>
Where: <file:line, config path, or dashboard/alert name>
Evidence: <exact code/config snippet or observed behavior>
Impact: <what fails during an incident or at scale, concretely>
Fix: <specific change, with code/config if short>
Effort: <S/M/L>

Conclude every audit with the two-question verdict: can on-call currently answer "why is this request slow?" and "what broke at 3am?" — YES/PARTIAL/NO, with the shortest path to YES.

Rules index

Top 10 non-negotiables

1. Every log line carries a trace/correlation ID. A log you cannot join to a request is gossip, not evidence.
2. ERROR means a human must act. If nobody should be woken or ticketed, it is WARN or below. Level discipline is alert discipline upstream.
3. No secrets or PII in telemetry — enforced at the logger/exporter, not by call-site vigilance. Redaction is infrastructure, not convention.
4. One wide event per unit of work (request/job/message) with outcome, duration, and business context — the canonical log line you grep at 3am.
5. Metric labels are bounded. No user IDs, emails, raw URLs, or free text. Cardinality explosions take down the monitoring you need most.
6. Never average percentiles. Aggregate histograms, then compute quantiles. A dashboard of avg(p99) is fiction.
7. OpenTelemetry API in libraries, SDK only at the edge. W3C traceparent propagated across every HTTP hop, queue, and async job.
8. Liveness checks process health only; readiness checks dependencies. Conflating them turns one slow dependency into a cluster-wide restart storm.
9. Every page is actionable, symptom-based, and runbook-linked. Alert on user pain (SLO burn rate, multi-window), not on causes (CPU, pod restarts).
10. Telemetry has a budget. Sample debug logs and traces deliberately (tail-sample to keep errors/slow), review cost monthly, delete signals nobody queries.