Failure

Here is a way of thinking no tool will hand you, because it is a stance, not a snippet. Most code is written as though it will succeed. The systems that actually survive are written as though everything will eventually fail, and the interesting question becomes not “how do I prevent errors?” but “how does the whole thing keep running when one part dies?”

The person who saw it first

Joe Armstrong was set a nearly absurd target at Ericsson: telephone switches allowed only a handful of minutes of downtime across decades of running (the famous “nine nines”, 99.9999999% uptime), while carrying millions of calls, being upgraded in place, on hardware certain to fail. He did the honest arithmetic: at that scale you cannot write software with no bugs, so aiming for bug-free is aiming at the wrong thing.

His flip was almost heretical. Stop trying to prevent every error; assume things will break and design so the system survives when they do. Isolate a failure so it cannot spread, detect it, restart that piece into a known-good state. He built the belief into Erlang and gave it a slogan that still startles people: “let it crash.” He was also a generous teacher who converted a generation through his book and talks, and his 2003 thesis laid it out formally. He died in 2019, widely mourned. See also The Actor Model.

Two philosophies, side by side

• C — defend by hand. Every call that can fail returns a code you must check: if (p == NULL), inspect errno, clean up, propagate. It is precise and fast, but fragile in a human way: one forgotten check is a crash, or worse, a silent corruption that surfaces hours later. Safety depends on the programmer never once being tired.
• Erlang — don’t defend; recover. Processes share nothing, so a failure cannot spread. A crashing process simply dies, and a supervisor whose only job is to watch restarts it. You write the happy path; the structure handles the sad one.

Why it endures

Strip away the telecom and this is a control loop, which is exactly the feedback-and-stability idea you already think in: measure the deviation (a process died), act to drive the system back to its stable state (restart it). That is why the idea outlived its original problem. Kubernetes liveness probes, retries, circuit breakers, “crash-only software” — all of them are rediscoveries of isolate-detect-restart. Learn it once, here, and you see it everywhere.

Try it

Install Erlang (brew install erlang), then run this. A worker crashes on purpose; its supervisor notices and brings the system back.

%%% "Let it crash": a worker dies, a supervisor restarts it.
main(_) ->
    process_flag(trap_exit, true),
    Worker = spawn_link(fun work/0),
    Worker ! boom,                       % provoke a crash
    receive
        {'EXIT', _Pid, Reason} ->
            io:format("worker died (~p), supervisor restarts it~n", [Reason]),
            _Fresh = spawn_link(fun work/0),
            io:format("system healthy again~n", [])
    end.

work() ->
    receive
        boom -> exit(kaboom);
        _    -> work()
    end.

Next in this thread: Concurrency (how Erlang makes a million of these processes cheap), Immutability (why Clojure removes a whole class of failure by making shared state impossible to corrupt), and Data modeling.

[!recall] “Let it crash” gives something up in order to gain something. What is the trade? It gives up defensive, per-error handling — you stop checking every return value on the spot. In exchange you get two things working together: isolation (shared-nothing processes, so a failure cannot spread) and supervision (a watcher whose only job is to restart the dead process into a known-good state). You write the happy path; the structure handles recovery.

Sources

• Armstrong, “Making Reliable Distributed Systems in the Presence of Software Errors” (2003 PhD thesis) — the whole philosophy, laid out at length and very readably.

See also: Erlang, The Actor Model, C, Clojure.