Learning AI #34

At one time, Knight Capital was the largest trader in US stocks, handling about 17% of all trading volume. One day, it deployed new trading software to the eight servers running its automated market-making system. (A market maker automatically matches up buy and sell orders of a stock.)

A technician copied the update to seven of the servers. The eighth server still had old, unused code left over from a system retired years earlier and never fully removed. Nobody double-checked the deployment of the new AI software that handled billions of dollars each day.

When the market opened, the old code on the eighth server started buying and selling shares on its own, firing off waves of orders nobody at Knight had actually placed. It was an outrageous error.

Automated alerts went out before the opening bell. Nobody acted on them. Trading staff watched their screens fill with activity they didn't recognize and spent the next 45 minutes figuring out what was happening and how to shut it off, server by server, while the algorithm kept firing.

By the time it stopped, Knight had made more than four million trades across 150 stocks that it was never supposed to trade. The loss came to more than $460 million, close to the firm's entire net worth, in less than an hour.

Knight survived that week only by taking an emergency cash injection from a group of investors, who took a controlling stake in exchange. It never recovered as an independent company. Within a year it merged with another company and the name Knight Capital stopped meaning anything on its own.

One unpatched server and one deployment nobody checked twice caused the whole thing, sitting quietly until the market opened and gave the old code something to do. Human-out-of-the-loop needed to be human-in-the-loop on an AI technology deployment so critical.

What stopped it was slow and manual. People stared at screens full of trades they hadn't placed, then pulled cables and killed processes one machine at a time, after the damage had already started.

The fix was people walking a data center floor.

The next time a company says its trading algorithm, or its AI system, has been tested and is “safe” to run unsupervised, trust, but verify. Knight's software had been tested too. The bug that took it down was a single server that missed an update. It ran for 45 minutes before a human fully understood what it was doing, and every one of those minutes cost real money.

Things I think about

Every astronaut who has actually looked, from Apollo crews to the ISS at about 250 miles up, says you cannot see the Great Wall of China with the naked eye.