- The Cautionary
- Posts
- Learning AI #28
Should a computer decide how long you sit in jail?
An 18-year-old named Brisha Borden grabbed an unlocked kid's bike off a lawn in Fort Lauderdale. She rode it a few yards, realized it was too small, and dropped it. She was arrested and charged with burglary and petty theft. Total value of the stolen goods: $80.
A 41-year-old named Vernon Prater walked out of a Home Depot with $86 in tools he hadn't paid for. Prater had previously served five years in prison for armed robbery, on top of several other theft convictions.
Both of them got run through COMPAS, a risk-assessment algorithm used across the US to predict who is likely to commit another crime. Judges see the score before setting bail. Some use it at sentencing and parole hearings too.
Borden, the teenager with the bike, scored an 8 out of 10. High risk. Prater, the career criminal with the armed robbery conviction, scored a 3 out of 10. Low risk.
The algorithm rated an 18-year-old who took a kid's bike as a bigger threat than a man who had already served time for armed robbery.
Two years later, journalists at ProPublica checked back on both of them. Bike-thief Borden had not committed another crime. Ex-con Prater was serving an eight-year sentence for breaking into a warehouse and stealing thousands of dollars in electronics.
The AI had it backwards.
Should an AI decide your time in jail?
That wasn't one bad prediction. COMPAS scores for more than 7,000 people arrested in Florida were tracked for two years post-arrest. Among those those that did not commit another crime in the two years, black defendants were rated high risk almost twice as often as white defendants. Huh?
There's a big problem sitting underneath the math. Nobody outside of Northpointe, the maker of COMPAS, knows exactly how COMPAS turns its intake data into a score. The company has treated its formula as a trade secret since the tool launched in 1998.
A Wisconsin man named Eric Loomis was sentenced to six years in prison partly on the value of his COMPAS score. He asked to see how the number was calculated and was denied. The U.S. Supreme Court declined to hear his appeal.
Eric Loomis never found out how his score was calculated. Neither did his lawyer.
It has been proven that judges relying on instinct and personal experience alone have been inconsistent and biased in their sentencing decisions, which is one of the reasons courts started using risk scores in the first place.
You are probably never going to be scored by COMPAS. But you may already be scored by something similar: a resume screener, an insurance pricing model, or a credit-approval algorithm.
A score decides something real about your life and the companies that build them call the formulae proprietary.
Ask who gets to see the math. If the answer is nobody, that should worry you more than the number comes back.
Things I think about
The human brain uses 20% of your body’s energy despite being 2% of its weight.