- The Cautionary
- Posts
- Learning AI #29
Several years ago, Allegheny County, Pennsylvania (home of Pittsburgh) started using AI software to decide which families were to be investigated for child neglect. The tool is called the Allegheny Family Screening Tool, or AFST.
Every time someone calls the county's child welfare hotline, the AI scores the child’s family “risk” from 1 to 20. Score high enough, and an investigation becomes mandatory.
A caseworker no longer gets to use judgment, understand the special circumstances that may exist, and wave it off.
The score comes from a mix of the family’s government records, including child welfare history, substance abuse treatment, mental health records, and jail and probation data. Buried in that mix is a marker for disability.
The tool checks whether any parent or child in the household receives SSI, the federal disability benefit, and tracks diagnoses like schizophrenia or mood disorders. Households with that marker score as higher risk.
A couple named the Hackneys brought their eight-month-old daughter to Pittsburgh Children's Hospital. She was having trouble feeding. Hospital staff found her dehydrated and malnourished and reported it to the county, which is standard procedure.
That report went into AFST algorithm. The Hackneys have an intellectual disability, a key variable the tool weighs. A social worker showed up, and their daughter went into foster care.
The Hackney’s attorney filed a federal civil rights complaint, arguing the tool used the Hackney’s disability against them. The complaint became one of several the US Department of Justice cited when it opened a review of AFST for likely discrimination against people with disabilities.
AI algorithms like AFST are pulling families apart without human intervention since they “score” the risk of leaving a child in the home.
The case gets worse from there. The people who built AFST decided, on purpose, to hide the numeric score from judges and from the parents' own attorneys in court. Their reasoning: if a judge sees a risk score, the judge will lean on the number instead of weighing the evidence.
So the two people with the power to challenge the algorithm's decision. the judge and the family's lawyer, never saw the number that triggered the case.
Carnegie Mellon University researchers later reviewed AFST's track record. Human social workers disagreed with the algorithm's risk assessment about a third of the time. By the time a caseworker can review and disagree with the decision made by the AI, disagrees, the family has already lost custody and is lost the the legal miasma of trying to get their kid back. An uphill battle to say the least.
Nobody at Allegheny County set out to punish disabled parents. The AI tool was built to catch cases of child neglect faster. Feed disability status into the AI model, and the model treats disability as risk since that was how the AI was trained. Disability status went into the tool, a computation was done, parents lose custody of a child, and then have to fight through the court system to get the child back. Something is broken.
Afterword
A judge ruled that the Hackneys have a fundamental constitutional right to care for their daughter and that the county's caseworkers and AI system "blatantly violated that right in a way that would shock the conscience of any reasonable person."
The Hackney’s now have a path to a fair fight in their upcoming trial.
Things I think about
By the end of this century, Africa is projected to hold 40% of the world’s population
*****