Safe Life, Fail Safe, and Damage Tolerance
I’m a fan of Mentour Pilot’s YouTube channel. His videos contain calm, clear explanations of why aircraft incidents happen, what could have been done differently to change the outcome, and what can prevent the incident in the future. His video on the 14 May 1977 Dan Air 707 crash had interesting ideas that can provide insight into IT.
In short, the right horizontal stabilizer of the aircraft failed and then broke off, resulting in the plane doing a back flip into the ground. There was nothing the pilots could have done – this was a design fault and a maintenance failure.
A Dan Air Boeing 707.
Source: Wikimedia
Safe Life
The first engineering design philosophy used by aircraft designers is “Safe Life.” The idea is simple: a component is designed to withstand the usage it will experience throughout its expected service life without developing dangerous wear or other failures.
Because no component lasts forever, Safe Life designs rely on retirement schedules. Parts are replaced before they are expected to fail. This concept is familiar to most people. Cars contain many components that are replaced based on age, mileage, or wear, such as tires, brakes, timing belts, and gaskets.
In IT terms, this concept translates to replacing parts before they are outdated, fail, or are compromised:
Hardware replacement schedules
Software update schedules
SSL certificate lifetimes
Key rotation
The advantage of Safe Life design is its simplicity. If the engineering assumptions are correct and the maintenance schedule is followed, the component should never fail in service.
The weakness is that the assumptions can be wrong. The de Havilland Comet was designed under the belief that the aircraft’s pressure vessel would outlast its useful service life. As it turned out, engineers did not completely understand how repeated pressurization cycles would affect metal fatigue, resulting in two fatal accidents and the reevaluation of how structures were designed and inspected.
Safe Life can fail in IT for the same reason it failed in aviation: the assumptions can be wrong. A server may fail years before its scheduled replacement date. A UPS battery may degrade faster than expected due to heat. A software vulnerability may be discovered long before the next planned upgrade cycle. Safe Life works best when the expected service life is well understood.
Fail Safe
The next engineering philosophy adopted by the aviation industry was the “Fail Safe” concept. This is the idea that a single failure should not cause catastrophic failure. The advantage of Fail Safe design is that a failed component becomes a maintenance issue rather than an immediate outage.
Fail safe should be a familiar concept to most IT staff:
Redundant network equipment, like high availability and failover
RAID arrays
Multiple domain controllers, server mirroring, distributed file systems
Dual power supplies
Multiple Internet connections
The Dan Air crash highlighted a weakness of the Fail Safe concept. Due to the design of the Boeing 707’s horizontal stabilizer, a fatigue crack formed in the rear spar at a location not anticipated by Boeing engineers and not inspected on normal maintenance checks. Because of this, the fail-safe design did not provide the protection Boeing engineers expected. On approach to Lusaka Airport in Zambia, the crack rapidly propagated through the remaining structure, causing a catastrophic loss of the right horizontal stabilizer.
Fail Safe can also fail in IT when the redundancy does not behave as expected. Two Internet connections may share the same upstream provider infrastructure. A pair of high-availability firewalls may depend on the same power source. A storage cluster of like disks may share the same firmware bug. What appears to be redundancy may still contain a hidden single point of failure.
Damage Tolerance
Inspections of the 521 similar Boeing 707 aircraft found 38 more aircraft with the same cracks, with four requiring complete horizontal stabilizer replacement.
The Dan Air crash helped drive a major shift in aircraft structural design. Engineers recognized that neither Safe Life nor Fail Safe design alone was sufficient. Damage could occur in unexpected locations, and inspections needed to be designed around the assumption that defects could already exist and that maintenance personnel need to find them before they became failures.
Damage Tolerance assumes that damage, defects, or degradation already exist and focuses on detecting and correcting them before they result in failure.
In IT terms, this translates to:
Hardware health monitoring
Vulnerability scans
Backup verification
Recovery tests
Resource monitoring
Log reviews
Safe Life, Fail Safe, and Damage Tolerance are not competing philosophies. Modern systems use all three. Components are replaced before they are expected to fail, critical services are designed with redundancy, and monitoring and testing are used to detect problems before they become outages.
The lesson for IT is the same as it is for aviation: no single approach is sufficient on its own. Replacement schedules can be based on incorrect assumptions. Redundancy can contain hidden points of failure. Monitoring is only effective if someone responds to the warnings. Reliable systems are built by combining all three approaches and understanding the limitations of each.