Researchers took 198 randomly sampled production failures from Cassandra, HBase, HDFS, Hadoop MapReduce and Redis, and read every one of them [R1]. In 84% of those failures, every event that triggered the failure had already been written to the log before the system went down [R3]. The median failure printed 824 log messages [R5].
Nobody was flying blind. They were reading the wrong 823 lines.
Inherit a running system, someone else's architecture, someone else's log statements, someone else's empty catch blocks, and that finding changes what to monitor in production first. The instinct is to buy collection. Collection was never the bottleneck. Deciding in advance which events would matter was.
What to monitor in production is the decision, made before the incident, about which events in a running system will be treated as signal: the four golden signals of latency, traffic, errors and saturation, plus the error handlers, the change surface, and the paths nobody thought to watch.
Key takeaways
The study is Yuan et al., presented at USENIX OSDI in October 2014 [R1]. Say the date out loud: twelve years old, and I couldn't find a comparable large-sample study of the same shape. That's not a reason to discount it. Its subject is error handling and logging discipline, and neither has been reinvented since. It is a reason to treat it as durable measurement rather than a fresh headline.
They pulled 198 failures at random from a pool of 17,216 user-reported failures across the five systems [R9], read the tickets and the patches, then manually reproduced 73 of them [R2]. They put the statistical precision on the record too: "under standard assumptions, the Central Limit Theorem predicts a 6.9% margin of error at the 95% confidence level for our 198 random samples" [R13].
Two findings carry this post. Finding 7: "For a majority (84%) of the failures, all of their triggering events are logged" [R3]. Finding 8: "Logs are noisy: the median of the number of log messages printed by each failure is 824" [R5]. And in 76% of the failures the system printed an explicit failure-related error message: an actual complaint about the thing that was wrong, up from 43% in the same group's earlier work on non-distributed systems [R4].
The 824 carries a caveat that cuts the wrong way for you. It was measured while reproducing failures under a minimal configuration and a minimal workload, excluding everything printed at start-up and shut-down [R5]. It's a floor, measured in a deliberately quiet laboratory. Your system, under real load with real tenants, is louder.
The authors' own summary is the least dramatic sentence in the paper and the most useful one: across this sample, the primary challenge was having to sift through relatively noisy logs [R5].
Scope, honestly stated: the sample is 198 user-reported failures drawn from five open-source, data-intensive distributed systems — Cassandra, HBase, HDFS, Hadoop MapReduce and Redis, so it is evidence about that class of system [R1]. Your inherited platform is probably a monolith and three services in a trench coat. What transfers is the mechanism, not the systems: the events were logged, the logs were noisy, and the handlers that should have acted on them didn't.
Of the 198 failures, 48 were classified catastrophic, meaning they stopped all or most users from using the system normally [R9]. Those 48 are where the paper gets uncomfortable.
Finding 10: "Almost all catastrophic failures (92%) are the result of incorrect handling of non-fatal errors explicitly signaled in software" [R10]. The software noticed. The software raised an error. Something downstream handled it wrongly, and the wrong handling — not the original fault — took the service down.
The contrast is the point. Among the non-catastrophic failures, only 25% involved incorrect error handling [R11]. The authors read that as evidence that error handling was mostly doing its job of stopping local errors from becoming outages [R11]. When it worked, you got a blip. When it didn't, you got the incident.
Then Finding 11, the one to bring to your next architecture review: "35% of the catastrophic failures are caused by trivial mistakes in error handling logic — ones that simply violate best programming practices; and that can be detected without system specific knowledge" [R12]. Break that 35% down and it's worse. Within the catastrophic failures this time — the denominator has flipped from the 25% in the paragraph above — 25% came from a handler that caught the explicit error and ignored it, 8% from aborting the system inside an over-caught exception, and 2% from a handler carrying a "TODO" or "FIXME" comment where the handling logic should have been [R12].
A comment. In production. Between your users and their data.
None of that needs a distributed-systems specialist to spot. It needs someone to read the error paths, which is exactly the work nobody schedules on a codebase they didn't write. That work belongs in a static review of an inherited codebase. This post is the runtime counterpart, for the handlers already carrying traffic.
The usual excuse for not reproducing an inherited system's failures is environmental: we can't stand up prod, so we can't chase this. The data doesn't support it.
Finding 3: "Almost all (98%) of the failures are guaranteed to manifest on no more than 3 nodes. 84% will manifest on no more than 2 nodes" [R8]. That's a second, different 84% — the share reproducible on two nodes, not the share whose triggering events were logged. The paper is full of numbers that rhyme; keep them apart.
And Finding 4: "74% of the failures are deterministic — they are guaranteed to manifest given the right input event sequences" [R7]. Not races. Not heisenbugs. Same sequence, same break. That's how 73 of the 198 got reproduced by hand [R2].
Three quarters deterministic, most visible on two nodes. Whatever is stopping you from investigating the failure modes of a system you inherited rather than built, it isn't cluster size. It's that nobody has been given the week.
Ask a CTO what takes their system down and you'll hear about faults: a dead disk, a dependency going away. The underweighted category is the change you approved.
In the Yuan sample, 23% of failures were caused by configuration changes, and of those, only 30% involved actual misconfigurations. The remaining majority "involve valid changes to enable certain features that may be rarely-used" [R6]. Valid. Correct. Reviewed, presumably. The setting was legal and the system fell over anyway, because a rarely-exercised path carried traffic for the first time.
The larger outage literature points the same way. Gunawi et al. analysed 1,247 headline reports and public post-mortems covering 597 unplanned outages at 32 popular internet services between 2009 and 2015 [R27]. Among outages with a reported root cause, UPGRADE was the largest single category at 16% of known causes, with CONFIG at 10%, and NETWORK and BUGS at 15% each [R30]. On the upgrades, the study is blunt about why testing didn't save anyone: "The developers had tested the software upgrades, but when the updates were pushed to the full ecosystem, they caused new failures/anomalies that were undetected in the offline testing" [R31].
One caution the study makes unavoidable: 355 of the 597 outages had UNKNOWN root causes [R30]. Every breakdown of why cloud services fail is computed over the minority somebody bothered to explain in public. Directional, not precise.
So config, feature flags and upgrades deserve the same instrumentation you'd give a dependency, and the same scrutiny you'd give the size of each change you ship. Not because operators are careless, but because valid changes light up code paths that have never carried traffic.
Start with the four golden signals — Google's prescription, not a measurement, then instrument what they don't cover on their own: the error handlers, the change surface, and the paths nobody thought to watch.
That first half is the advice everybody gives you. Know exactly what kind of advice it is.
The golden signals come from Chapter 6 of Google's Site Reliability Engineering book, "Monitoring Distributed Systems", written by Rob Ewaschuk: "The four golden signals of monitoring are latency, traffic, errors, and saturation. If you can only measure four metrics of your user-facing system, focus on these four" [R14]. Latency is "the time it takes to service a request"; traffic, "a measure of how much demand is being placed on your system"; errors, "the rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly …"; saturation, "how 'full' your service is" [R15].
That is Google's prescription. It is not a measurement: no sample, no method, no dataset — a very good engineer at a very large company telling you what to do. Useful provenance, not evidence. The evidence here comes from Yuan, Gunawi, Niedermaier and Davidovič. Mixing the two is how posts end up citing "research" that was always an opinion.
Two other frameworks get muddled with the golden signals constantly, and they are neither the same thing nor by the same people. The RED Method is Tom Wilkie's, from 2015: Rate, the number of requests per second; Errors, the number of those requests that are failing; Duration, the amount of time those requests take [R20]. It's scoped to services. The USE Method is Brendan Gregg's, scoped to resources: "For every resource, check utilization, saturation, and errors", where resource means "all physical server functional components (CPUs, disks, busses, ...)" [R19]. Wilkie draws the line himself: "The USE Method doesn't really apply to services; it applies to hardware, network disks, things like this" [R20]. In that same write-up the golden signals get called "basically the same as the RED Method, but includes saturation" [R20]. A fair bridge, but Grafana's characterisation, not Google's.
Golden signals by Google's prescription, RED for services by Wilkie, USE for resources by Gregg. Don't let a vendor sell you one as all three. Then add what the failure data says those signals won't catch on their own.
Instrument the error paths, not just the error rate. If 92% of catastrophic failures came from mishandling errors the software had already signalled [R10], a swallowed exception is a monitoring gap, not a style nit. Emit on the handler, not only the request.
Instrument the paths nobody thought to watch. Gunawi's list of undetected failures includes one case worth laminating: "load spikes of authentication requests (that were not monitored, while spikes of read/write requests were monitored)" [R29]. Alongside it: memory leaks that eventually crashed the whole system, expiring certificates that piled up into backlogs, grey partial hardware failures, corrupt configurations [R29]. The failure gets in through the door you didn't put a sensor on.
And don't assume redundancy covers you. The same authors: "although the failover code that will activate the redundancies is ready to run, it will sit idle if the root failures are not detected or anticipated" [R28]. Avoiding single points of failure, they conclude, "is not merely about redundancies, but also about the perfection of failure recovery chain: complete failure detection, flawless failover code, and working backup components" [R28].
Then bound the alerting, or you've rebuilt the 824-message problem with a pager attached. Ewaschuk's rule: "every page should be actionable", and "if a page merely merits a robotic response, it shouldn't be a page" [R16]. His reason is human: "I can only react with a sense of urgency a few times a day before I become fatigued" [R16]. Chapter 11, by Andrea Spadaccini, puts a ceiling on it: "the maximum number of incidents per day is 2 per 12-hour on-call shift", because handling one incident end to end takes about six hours [R17]. Again, a prescription, not a measurement of what teams actually live with.
To judge the alerting itself, the SRE Workbook's four attributes beat a dashboard count: precision, "the proportion of events detected that were significant"; recall, "the proportion of significant events detected"; detection time; reset time [R18].
One finding here isn't technical at all. Niedermaier et al. interviewed 28 software professionals about observability and found the constraint they reported was organisational rather than a lack of tooling: a "discrepancy in the awareness regarding the importance of the topic, both from the management as well as from the developer perspective", and "a strong need for an organizational concept including strategy, roles and responsibilities" [R32]. No vendor sells you out of that. Somebody has to own it.
Every observability vendor's case study ends the same way, with a recovery-time improvement. The most rigorous published look at that metric came out of Google, and it concludes the metric can't carry the claim.
Štěpán Davidovič's "Incident Metrics in SRE: Critically Evaluating MTTR and Friends" (O'Reilly, March 2021, published on Google's SRE site) ran Monte Carlo simulations over real incident data [R21]. The data came from three public incident dashboards at internet companies of roughly one to two thousand employees. Their mean recovery times were 2h 26m, 2h 31m and 4h 31m, against standard deviations of 5h 16m, 5h 1m and 6h 53m [R24]. Note the shape of that. The spread is twice the average.
He then simulated a product that genuinely shortened every incident by 10%, drew two equal samples from the real duration distribution, and repeated it 100,000 times. Result: "Even though in the simulation the improvement always worked, 38% of the simulations had the MTTR difference fall below zero for Company A, 40% for Company B, and 20% for Company C" [R22]. A real improvement on every incident, and MTTR still moved the wrong way in a fifth to two fifths of trials.
The reverse test is worse. With nothing changed at all, "there's a 19% chance that there is a half-hour improvement (or better) of MTTR in Company A (and 23% for Company B, and 10% for Company C)...even though in this simulation, you did not change anything about the incidents" [R23]. Half an hour of pure noise, available on demand, at roughly one-in-five odds.
His conclusion: "I have demonstrated that even in a favorable analysis setup, MTTx cannot be used for many practical purposes where it has been advertised to be useful, such as evaluating reliability trends, evaluating results of policies or products, or understanding the overall system reliability" [R21]. The cause is structural: "the typically low volume of incidents and high variance of their durations". And he extends the warning explicitly: "Many other incident metrics, such as the time to detection, suffer from the same problem" [R25].
That last line applies to what this post is recommending, so take it seriously. It doesn't mean don't instrument detection. Davidovič also recommends the narrower question: "An example of this might be measuring and studying the time to detection specifically, or time spent on some common incident-response activities" [R26]. Study your detection gaps case by case. Don't publish an average and call it a trend.
DORA closes the loop from the other side. It separates the terms. Monitoring "allows teams to watch and understand the state of their systems"; observability "allows teams to actively debug their system" [R33]. And it states its observability finding purely qualitatively: comprehensive monitoring and observability "positively contributes to continuous delivery" [R34]. No percentage, no effect size. None of DORA's five core metrics measures monitoring, detection or observability; they measure delivery and instability [R35].
DORA also retired mean time to restore in 2023, replacing it with failed deployment recovery time, because "previous definitions did not distinguish between a failure initiated by a software change and a failure caused by external factors, such as a data center outage" [R36]. Anyone quoting DORA for an observability-and-MTTR number is quoting a metric DORA stopped using, for a finding DORA never sized. (Which engineering metrics genuinely predict delivery is a different question with a different answer.)
Even DORA's strongest statement here stays unquantified: when failures are "primarily reported by a source external to the operations team, such as by the network operations center (NOC) — or worse, by customers — rather than internal monitoring, performance suffers" [R37]. Directionally right, no number attached. Be suspicious of anyone who supplies one.
So justify the work without a metric. Justify it with the last three incidents, and the log lines nobody read.
This work isn't hard. Every task in it is small, well-specified, and permanently outranked by whatever is on the roadmap. Add structured context to the payments error handler. Alert on authentication load, not just read and write. Emit a metric when a feature flag flips. Not one of them is a big job, and together, they're the difference between an incident you can read and 824 lines you can't.
That's the gap Dev On Demand is built for: 1 dedicated AI-augmented engineer on your queue, a 3-day task cycle with daily async updates, and a task-by-task approval gate so you see each piece before the next one starts. Single stream $3,495/mo, dual stream $6,795/mo, engineer assigned the same business day, 5-day first ship. There's a 5-day replacement guarantee if the fit is wrong, and you can cancel any time, no lock-in, no notice period required. Proof of Quality is one real task, you judge the engineer before subscribing. Instrumentation is a good first task because it's easy to grade: either the next incident is legible or it isn't.
Whether you use us or not, do this one thing this week. Take your three most recent production incidents, find the log line that recorded the triggering event, and check its timestamp against when a human first noticed. If the line is there — and the research says it usually is — you don't have a data problem. You have a decision nobody made about what would matter.
Start with the four golden signals of latency, traffic, errors and saturation [R14], then instrument what they don't cover on their own: the error handlers, the change surface, and the paths nobody thought to watch. In the Yuan et al. sample, 92% of catastrophic failures came from incorrect handling of errors the software had already signalled [R10], and 23% of all failures were caused by configuration changes [R6].
They are three different frameworks by three different authors. The four golden signals of latency, traffic, errors and saturation are Google's prescription, written by Rob Ewaschuk in Chapter 6 of the Site Reliability Engineering book, and they are advice rather than a measurement [R14]. The RED Method is Tom Wilkie's, from 2015, and is scoped to services: rate, errors and duration of requests [R20]. The USE Method is Brendan Gregg's and is scoped to resources: for every resource, check utilization, saturation and errors [R19].
Usually not. In the Yuan et al. sample, 98% of the failures are guaranteed to manifest on no more than 3 nodes [R8], and 74% of the failures are deterministic, meaning they are guaranteed to manifest given the right input event sequences [R7]. That is how 73 of the 198 failures were reproduced by hand [R2].
The study was presented at USENIX OSDI in October 2014, and its sample was 198 user-reported failures drawn from five open-source, data-intensive distributed systems — Cassandra, HBase, HDFS, Hadoop MapReduce and Redis — so it is evidence about that class of system [R1]. Its subject is error-handling and logging discipline, so treat it as durable measurement rather than a fresh headline. Its headline number, that in 84% of failures all triggering events were already logged [R3], describes a mechanism, not a product generation.
No. Simulating a product that shortened every incident by 10%, Štěpán Davidovič found the MTTR difference still fell below zero in 38% of the simulations for Company A, 40% for Company B and 20% for Company C [R22]. His conclusion is that MTTx "cannot be used for many practical purposes where it has been advertised to be useful, such as evaluating reliability trends, evaluating results of policies or products, or understanding the overall system reliability" [R21]. DORA retired mean time to restore in 2023, replacing it with failed deployment recovery time [R36].
