The Interlock Nobody Missed - An RC Inspired by IIM Indore

Between 1985 and 1987 six patients in the United States and Canada received massive overdoses of radiation from a cancer therapy machine called the Therac-25, and at least three of them died. The machine was made by Atomic Energy of Canada Limited and was meant to be an improvement. It was smaller, cheaper and controlled almost entirely by software.

Its predecessors, the Therac-6 and the Therac-20, had carried hardware interlocks, physical devices that stopped the beam if something went wrong. The designers of the Therac-25 removed them, reasoning that software could do the same job. They also reused code from the older machine. Investigators later found that this code already contained the same fault, which the old hardware had been silently catching for years.

The fault was a race condition. If an experienced operator edited the treatment settings quickly, the software could leave the machine in a dangerous mode while the screen showed that everything was in order. The only warning the operators saw was an error labelled Malfunction 54, which the manual did not explain. The message appeared often, and treatment could be resumed with a single keystroke, so operators learned to treat it as noise.

Responsibility dissolved along the way. The manufacturer told hospitals that an overdose was impossible, and when patients were injured at a clinic in Tyler, Texas, in 1986, it took a hospital physicist to reproduce the fault by working at the speed of the operators. The makers had trusted their code, the operators had trusted the machine, and the regulators had trusted the makers. Each party held a reasonable belief, and no one was assigned the job of doubting.

Each party held a reasonable belief, and no one was assigned the job of doubting.

Defenders of the manufacturer note that software can never be tested completely and that the company did investigate each report and revise its code. Both points are true. But they miss what actually changed. A bug is a technical failure, and every large program has them. Deleting the independent check at the moment the stakes rose was a decision, and treating each complaint as an anomaly followed from a design that assumed safety instead of testing for it.

The question has only grown. Software now replaces physical safeguards in cars, aircraft and hospitals. When a system fails although every component did what it was built to do and every person did what the role required, the harm is real but the wrongdoing is hard to find. Ethics usually begins with someone who acted badly. This case begins with a machine in which no one clearly did, and it is unclear whether responsibility can be given to a system at all or only to the people who decided what the system would be allowed to doubt.