Los Angeles County, home to the nation’s largest locally-run child welfare system, is grappling with how to measure and respond to the risk that a child will be abused.
At the heart of this confounding, complicated issue are two hard-to-answer questions.
The first is whether it’s time to move forward with “predictive risk modeling,” which uses huge data sets and advanced algorithms to discern the likelihood a child will be abused.
The second, and more important, question: how to respond when risk assessments suggest that a child will be maltreated in the future, but is not in imminent harm today?
Last week, Los Angeles County’s Office of Child Protection (OCP) issued a long-awaited report on these issues. Spurred by the death of an 11-year-old Echo Park boy named Yonatan Aguilar last summer, the office was charged with weighing the county’s current risk-assessment tools against the emerging practice of “predictive analytics.”
Beyond navigating the ethical and practical minefield of what tools should be used to determine risk, the OCP, in its most academic report to date, dove into the fraught question of what to do when a family is deemed at high risk, but there is no legal basis to intervene.
The answer to that question is difficult for child protection agencies across the country. On the one hand, better risk data offers systems the opportunity to target in-home services to the families who most need them.
On the other hand are cases like Yonatan’s: A child left at home dies tragically, the media catches wind, and policymakers and advocates are quick to insist that child protection officials should have done more.
Doing more requires having risk assessment tools that are reliable, alongside a willingness to be more aggressive in responding to that risk. On both counts, in L.A. and across the country, there is no straightforward answer.
L.A. County’s Risk Detection System
Currently the county’s Department of Children and Family Services (DCFS) uses a suite of risk assessment tools to augment their decision-making when responding to a referral of child abuse. L.A. County and jurisdictions across the country use something called Structured Decision Making (SDM), developed by the National Council on Crime and Delinquency (NCCD), at critical points during a child abuse investigation.
These tools are essentially questionnaires that social workers fill out. Based on what they fill in, SDM will shoot out guidance to the worker on what to do next. But the social worker and his or her supervisor have the ultimate decision about whether to follow those instructions or override them.
Workers first use SDM at the DCFS child abuse hotline, a sprawling call center on the outskirts of downtown L.A. Hotline staff use SDM’s “intake assessment” to help them decide whether to send out a child abuse investigator.
Once in a family’s home, the worker uses a “safety assessment” to help decide whether there is imminent danger that a child will be harmed. In such a scenario, DCFS will remove the child.
After determining safety, a DCFS investigator often conducts a “risk assessment” meant to gauge the likelihood that children in the home will be referred to the department again in the next 12 to 18 months. The four possible designations are “low,” “moderate,” “high” and “very high.”
In 2015, DCFS conducted nearly 70,000 safety and risk assessments involving more than 145,000 children. Nearly 21,000 of those risk assessments were rated “high” or “very high.”
In the case of Yonatan, who was found dead in a closet in his mother’s Echo Park home in August 2016, the discrepancy between safety and risk assessments sparked the County Board of Supervisors to order the report submitted by the OCP last week.
From 2009-2012, Yonatan’s family’s was assessed by DCFS social workers four times. The SDM safety assessments conducted during that time showed no threats to Yonatan’s safety, while risk scores was always rated “high.” Given that the risk assessments were focused on future maltreatment, and Yonatan was not in imminent danger, DCFS social workers closed the case.
“It appears that the SDM risk assessment tool got it right,” the OCP report states. “The questions remain: what action did DCFS take, or what action should it have taken?”
Responses to Risk
In Yonatan’s case, the allegations of abuse were never substantiated. This is important because a substantiation of abuse, coupled with a clear and present danger to a child, form the legal basis for removing a child.
While the OCP report points to varying guidance on what to do in such a scenario, it references DCFS policy stating:
“Under no circumstances are users to select the client disposition of ‘Open New Case’ for a child without a substantiated allegation.”
This directly contradicts instructions in the SDM manual, which recommends opening a case in almost all circumstances involving “high” or “very high” risk referrals, even when the allegation is inconclusive or unfounded.
As mentioned earlier, nearly 21,000 referrals were deemed “high” or “very high” risk in 2015. Given that the average referral that year included at least two children, it is safe to assume that about 40,000 children were deemed in high risk of subsequent maltreatment. Only 26,701 children were substantiated victims of child abuse in 2015. So it stands to reason that a significant number of high-risk cases were closed, and that adopting a policy that aligns with SDM’s guidance would bring more children into the system.
Kathy Park, CEO of NCCD, says that jurisdictions across the country have policies that “promote” unfounded or inconclusive cases that have high or very high risk scores.
By focusing on these risky cases, Park says child welfare agencies can better determine “the population that is most likely to benefit,” from additional services and surveillance from the system.
The OCP report calls on DCFS and county officials to tackle this question head on.
“DCFS’ policy on how to respond when an SDM risk level is high or very high, but referral allegations are unfounded or inconclusive, should be revised to include case promotion and the appropriate level of manager approval needed prior to referral closures, as well as further assessment of the family’s needs for connections to appropriate voluntary services and/or community supports,” it reads.
But attempts to provide voluntary services to families after a referral of abuse has been lodged have had mixed results. Most recently, the field has grappled with the successes and failures of a policy orientation called differential response, which bumps lower-risk cases into a voluntary track.
OCP Executive Director Michael Nash and Assistant Executive Director Carrie Miller say that they don’t want to see current policy in L.A. change per se, but rather see heightened risk as warranting more more voluntary services.
“We are only suggesting that in cases where there is a ‘high’ or ‘very high’ risk for re-referrals to the system, it would be helpful to the family to take a preventative approach and offer to connect them to services and supports that could help to minimize this risk,” Miller, who took the lead on drafting the report, said in an email. “Since they [the services] would be voluntary, the family has the complete right to refuse them, and that’s OK. But at least we will know that we tried to help.”
Another focus of the report was to weigh the relative strengths of SDM against predictive analytics. Predictive analytics in child abuse risk assessment hinges on aggregating data sets to create risk profiles, much like SDM has done for decades.
The key difference is that, unlike SDM, predictive analytics tools would not be subject to “operator error” associated with filling out a questionnaire. It would also save considerable time, which proponents argue is critical to free up workers to do actual social work.
But, as the report points out, L.A. is far from being ready to implement a predictive analytics tool.
In 2014, the county concluded a pilot program with SAS, the largest data analytics firm in the world. Dubbed project AURA, the predictive analytics experiment was able to discern the children most likely to have a critical incident or die at the hands of their parents, but also flagged many children who suffered no such fate.
“At this point, SDM is the only game in town and we have to use it correctly,” Nash said.
That entails overcoming some of the major hurdles that have dogged the system here and in other jurisdictions.
The OCP report reviewed 726 cases of a near-fatal or fatal incidents stemming from maltreatment in the years spanning 2011 to 2015. Of those cases, SDM was mentioned in 17 percent of the critical incident/child fatality reports furnished by DCFS. In two thirds of the cases where SDM was included, it was noted because there was at least one problem with its application.
This finding was consistent with a 2012 Children’s Special Investigative Unit report that found improper use of SDM contributed to nine out of 14 child fatalities and one critical incident.
In addition to a data review, the OCP held focus groups with social workers, some of whom said that they “use it [SDM] because they ‘have to,’ not because they believe it is helpful or adds value to their work with families.”
According to the report, “several caseworkers also reported that SDM was not used so much to guide the decision making process, as more frequently, simply to document decisions after they had already been made.”
Of the report’s five recommendations, three are focused on improving the county’s use of SDM, predominantly through increased oversight and training.
Brandon Nichols, acting DCFS director, said more attention needs to be devoted to proper implementation of SDM in the short term.
“As long as we have it, we need to use it correctly,” Nichols said. But he added that he didn’t see the system as “a long-term investment.”
“Will predictive analytics ultimately replace SDM?” Nichols said. “I think yes.”
When it comes to predictive analytics, the report states that “the progress of predictive analytics should be monitored to determine its readiness to be considered for use in Los Angeles County.”
It also pointed to the need to adopt “standards” before deploying any predictive analytics tool in the county.
“It’s not an earth-shattering recommendation, but it’s the bottom line,” Nash said.
What to Do About Risk?
To date, a determination of safety has been the deciding factor in whether to remove a child. But when it comes to offering services that will prevent a child from coming to the attention of the system later on, risk is of critical importance.
Nichols agreed with the OCP’s conclusion that DCFS can do better in working with high-risk families.
“I am in agreement with the recommendation that on unfounded cases we need to be taking a second look, if SDM is telling us it is a risky situation,” Nichols said.
In such a scenario DCFS can, and often does, offer voluntary services, which a family in need may deny. The reasons for denying services are myriad, but fear of deeper involvement with the system is clearly one.
Inasmuch, child welfare systems across the county find themselves in a Catch-22. They are worried that something will happen to a child, but cannot reliably ensure that a family will access the support it needs to mitigate that risk.
While the OCP report did not recommend putting more weight on risk, it did ask the field to have a discussion about its importance.
So these are the questions.
If we see more cases like Yonatan’s, where risk was known, but the worst still happens, does the field need to act more aggressively to mitigate that risk?
If the answer is yes, are voluntary services adequate?
If the answer is no, then there is a heavy moral question at hand: Should perceived risk to a child trump the integrity of a family?
In the near term, DCFS, in reaction to the OCP report, will work to improve its use of SDM. Eventually, a predictive analytics model will overtake SDM as the preferred method of detecting risk.
In both scenarios, the county will become more adept at homing in on the children most likely to suffer abuse.