DevOps Observability Interview Questions — Practice Quiz

The three pillars are logs, metrics, and traces, and each answers a different question. Metrics are cheap aggregate time-series — great for 'is latency up?' and dashboards/alerts, but they aggregate away the per-request path, so they tell you that something is slow, not where in a multi-service call. Logs are discrete events rich in detail, but on their own they are per-service and don't automatically reconstruct one request's journey across services. Distributed tracing is purpose-built for this: a trace ties together the spans of a single request as it crosses service boundaries, each span timed, so you can read off exactly which downstream hop dominates the latency. The claim that a single latency gauge tells you exactly which service is slow overstates what a single metric can do; the claim that grepping each service's log lines reconstructs the full causal request path automatically assumes logs auto-correlate across services (they don't without a shared trace/correlation id); the claim that alerting is the fourth pillar and pinpoints the slow hop directly is wrong — alerting is built on metrics/logs, it is not a fourth pillar and does not localize a slow span.

Line B is structured logging: a machine-readable object with named, typed fields. A log platform parses it into indexed fields (user_id as a number, amount as a number), so a query like event:checkout AND amount > 100 is exact and survives wording changes. Line A is a human sentence — to extract the amount you'd need a fragile regex that breaks the moment someone changes 'checked out for $' to 'purchased', drops the dollar sign, or localizes the message, and the value arrives as text, not a comparable number. That brittleness is exactly why production systems standardize on structured logs with consistent field names (and a correlation/trace id) across services. The claim that free-text logs are easier for machines to query has it backwards; prose is easy for humans, hard for machines. The claim that structure makes no difference is wrong because structure is precisely what makes reliable field queries possible. The claim that numeric thresholds can never be evaluated from logs is false — you absolutely can threshold over a parsed numeric log field; metrics are often derived from such logs, but the log field itself is queryable.

In Prometheus alerting rules, the for field defines a duration that the rule's expression must keep returning results before the alert is considered firing. Until that duration elapses, the alert stays in the pending state, which filters out brief blips and reduces noisy pages.

This is foundational SRE vocabulary, and interviewers listen for the stack order rather than just the definitions in isolation: measure (SLI) → set a target on the measurement (SLO) → derive operational slack from the target (error budget). A strong answer also grounds it with a concrete numeric example (a 99.9% SLO on 1,000,000 requests over 28 days permits ~1,000 failures) and names the operational consequence of the budget — it isn't just an accounting artifact, it's the number that decides whether the team ships fast or slows down to firefight. Candidates who define all three correctly but can't say how spending/exhausting the budget changes team behavior are missing the practical half of the concept.

This is the context-propagation mechanism that OpenTelemetry describes as the core concept enabling distributed tracing at all. It starts when a request arrives with no existing trace context, so the first service acts as the entry point: it mints a new trace ID and opens the root span. Before that service calls anything downstream, it serializes its current span's identifiers into the outgoing request — standardized today as the W3C Trace Context 'traceparent' header — so the propagation format is consistent across languages and vendors. The receiving service extracts that header, and instead of starting an unrelated new trace, it opens a child span that records the caller's span ID as its parent, which is what stitches the two services' work into one causal chain (this inject/extract pair repeats at every hop in a multi-service call). Each service exports its own finished spans, independently and typically batched, to a collector or backend — export doesn't wait for the whole request to finish. Only at the backend does assembly happen: every span sharing the trace ID gets grouped and ordered by its parent/child links into the waterfall view engineers actually read to see which hop dominated the latency. Getting inject/extract in the wrong order relative to the call, or forgetting that assembly is a backend-side step rather than something each service does locally, are the common misconceptions this ordering surfaces.

This is a senior-mid design question testing whether the candidate can turn a vague reliability goal into a concrete, decision-ready SLO contract. A strong answer defines the SLI as a good-to-valid ratio with explicit exclusions, picks a measurement point that approximates user experience, converts the SLO target into a real budget with arithmetic, attaches a policy with a named decision owner, and uses multi-window burn-rate alerting instead of raw error thresholds. The rubric weights the SLI definition and budget math highest because they determine whether the SLO is measurable and meaningful at all.

An error budget is derived directly from the SLO: it is 100% minus the objective, so a 99.9% target permits 0.1% unreliability over the measurement window. That budget is a currency. While it is unspent the team has room to move fast — deploy often, take calculated risk — and when it runs out, the agreed error-budget policy kicks in: stop shipping risky changes and redirect effort to reliability until the service earns budget back. Its real organizational value is making the eternal velocity-vs-stability tension an objective, shared number instead of a turf war between developers who want to ship and ops who want stability. The claim that the goal is always to keep the error budget fully unspent is the classic trap: a budget that is never spent means your SLO is too loose (or you're over-investing in reliability users don't perceive) — you're leaving velocity on the table, so consistently spending zero is a signal to recalibrate, not a victory. The claim that the error budget is set by counting the number of incidents per quarter, independent of any SLO is wrong: the budget is computed from the SLO/SLI math, not by tallying incident counts.

False — correlating a log line with the active trace requires the OpenTelemetry logging signal or bridge to read the active span's context and inject trace_id/span_id into the emitted record; it doesn't happen just because tracing and logging code run in the same process. In practice this is often lightweight — the .NET SDK enables logs-to-activity correlation with no extra user code once OpenTelemetry logging is wired up, and Python needs an environment variable (OTEL_PYTHON_LOG_CORRELATION=true) plus adding %(otelTraceID)s/%(otelSpanID)s to the log formatter — but it is always a deliberate integration step. A bare console.log/print call bypasses that pipeline entirely and emits an ordinary line with no trace fields, regardless of which span happens to be active at that moment. This is exactly why 'log-trace correlation' is documented as its own setup step by every OpenTelemetry language SDK and observability vendor, rather than being an automatic side effect of co-located code.

This question separates candidates who've only used a metrics dashboard from those who understand how a time-series database actually stores data. The interview point is the mechanism: a metric's identity is its name plus its full label set, so cardinality is combinatorial — each additional label multiplies the series count by the number of distinct values that label can take, not adds to it. A strong answer names the mechanism (unique label-value combination = new series = new memory/storage cost), gives the canonical bad examples (user_id, request_id, session_id, raw/unnormalized URLs), contrasts them with safe bounded labels (status_code, method, region), and states the production consequence (memory exhaustion, storage growth, slow queries, and the operational trap of teams dropping retention right when they need the history most). The fix — keep unbounded fields out of metric labels and use logs/traces for per-entity lookups — shows the candidate understands why the three observability signals are architecturally different tools, not interchangeable ones.

The bug is cardinality explosion, and it's caused by the path label. req.originalUrl is the raw, unnormalized URL — it includes path parameters (/users/48213/...) and the query string, so nearly every request produces a URL that has never been seen before. Because a Prometheus time series is identified by metric name plus its full label set, every one of those distinct path values creates a brand-new, permanent time series that has to be indexed and held in memory — so memory grows roughly linearly with traffic instead of being bounded by the small number of actual endpoints the service exposes. The fix is to label with the matched route template the router resolved the request to (Express exposes this as req.route.path, giving a bounded value like /users/:id/orders/:id regardless of which user or order is requested) and to never include the query string in a label at all. The general rule this illustrates: any field whose value-space scales with users, requests, or time — IDs, session tokens, raw URLs, full timestamps — does not belong in a metric label; that kind of high-cardinality lookup is exactly what logs and traces are architected for, while a metrics store is architected for bounded aggregation. The claim that switching to res.on('close', ...) fixes the unbounded memory growth is a red herring — the finish-vs-close event timing has nothing to do with unbounded label growth. The claim that Prometheus counters leak memory by design and that calling .reset() on the counter on a timer fixes it misdiagnoses the failure as a client-library leak and proposes periodically discarding real data instead of fixing the label; it wouldn't even work, since new unique paths would keep arriving between resets and the underlying cause is untouched. Moving new prom.Counter(...) inside the app.use callback makes it worse: creating a new Counter object per request breaks the entire point of a counter (a single accumulating series) and does nothing about the unbounded path values driving the memory growth.