Widely Used Health Care Prediction Algorithm Found to Be Biased Against Blacks

From predicting who will be a repeat offender to who’s the best candidate for a job, computer algorithms are now making complex decisions in lieu of humans. But increasingly, many of these algorithms are being found to replicate the same racial, socioeconomic or gender-based biases they were built to overcome.

This racial bias extends to software widely used in the health care industry, potentially affecting access to care for millions of Americans, according to a new study by researchers at the University of California, Berkeley, the University of Chicago Booth School of Business and Partners HealthCare in Boston.

The new study, published Oct. 25 in the journal Science, found that a type of software program that determines who gets access to high-risk heath care management programs routinely lets healthier whites into the programs ahead of blacks who are less healthy. Fixing this bias in the algorithm could more than double the number of black patients automatically admitted to these programs, the study revealed.

“We found that a category of algorithms that influences health care decisions for over a hundred million Americans shows significant racial bias,” said Sendhil Mullainathan, the Roman Family University Professor of Computation and Behavioral Science at Chicago Booth and senior author of the study.

“The algorithms encode racial bias by using health care costs to determine patient ‘risk,’ or who was mostly likely to benefit from care management programs,” said Ziad Obermeyer, acting associate professor of health policy and management at UC Berkeley and lead author of the paper.

“Because of the structural inequalities in our health care system, blacks at a given level of health end up generating lower costs than whites,” Obermeyer said. “As a result, black patients were much sicker at a given level of the algorithm’s predicted risk.”

By tweaking the algorithm to use other variables to predict patient risk, such as costs that could be avoided by preventative care, the researchers were able to correct much of the bias that was initially built into the algorithm.

“Algorithms by themselves are neither good nor bad,” Mullainathan said. “It is merely a question of taking care in how they are built. In this case, the problem is eminently fixable — and at least one manufacturer appears to be working on a fix. We would encourage others to do so.”

More generally, said Obermeyer, incorporating routine audits into algorithm developers’ workflows would help. “For algorithms, just as for medicine,” he said, “we’d prefer to prevent problems, instead of curing them.”

Digging up the roots of algorithmic bias

Uncovering algorithmic bias — be it in the criminal justice system, in hiring decisions or in health care — is often hindered by the fact that many of the prediction algorithms in use today are designed by private companies and proprietary, making it difficult for data scientists and researchers to analyze them.

To tackle this problem, Mullainathan and Obermeyer partnered with researchers at an academic hospital that was using a risk-based algorithm to determine which patients were getting preferential access to a high-risk care management program. Programs like this are designed to improve care for patients with complex medical needs by providing them with additional attention and resources.

For 43,539 white patients and 6,079 black patients enrolled in the hospital, the researchers obtained the algorithm-predicted risk score and compared it to more direct measures of a patient’s health, including number of chronic illnesses and other biomarkers.

They found that, for a given risk score, blacks had significantly poorer health than their white counterparts.

“Instead of being trained to find the sickest, in a physiological sense, [these algorithms] ended up being trained to find the sickest in the sense of those whom we spend the most money on,” Mullainathan said. “And there are systemic racial differences in health care in who we spend money on.”

Patients whose risk scores landed in the top 97% were automatically identified for enrollment in the care management program. By correcting for the health disparities between blacks and whites, the researchers found that the percentage of black people in the automatic enrollee group jumped from 18% to 47%.

But there is room for hope, Obermeyer said. Training the algorithm to determine risk based on other measurable variables, such as avoidable cost, or the number of chronic conditions that needed treatment in a year, significantly reduced the racial bias.

And when alerted to the bias built into its algorithm, the software manufacturer was very motivated to address the issue, Obermeyer said.

“Algorithms can do terrible things, or algorithms can do wonderful things. Which one of those things they do is basically up to us,” Obermeyer said. “We make so many choices when we train an algorithm that feel technical and small. But these choices make the difference between an algorithm that’s good or bad, biased or unbiased. So, it’s often very understandable when we end up with algorithms that don’t do what we want them to do, because those choices are hard.”