A study by two University of Michigan researchers found errors in a Pennsylvania public benefits screening tool that could have wrongly told people they were not eligible for benefits, when in fact they were.

The researchers tested an online screening tool with tens of thousands of hypothetical family scenarios to see if the tool correctly or incorrectly predicted their eligibility for public assistance programs.

The tool incorrectly told the researchers’ test cases that they were not eligible for a state child care subsidy in every instance, when thousands should have been considered eligible, according to the paper.

The study was published in an academic computer science journal, Proceedings of the ACM on Human-Computer Interaction.

Advertise with us

State human service officials say they are looking into any potential errors raised by the study and will correct any errors “as swiftly as possible,” but advocates say this illustrates a larger problem with automating some government processes during a time of economic hardship.

With Pennsylvania facing double-digit unemployment, more people are applying for programs such as food stamps and medical assistance, and with the state’s county assistance offices still closed to the public, many are doing so online.

Statewide, enrollment in Medicaid is up more than 178,000 people (6.3%) since February. Enrollment in the Supplemental Nutrition Assistance Program, commonly called food stamps, is up 101,500 people (5.8%) since February, according to state statistics. More than 3 million Pennsylvanians are enrolled in Medicaid and more than 1.9 million in SNAP.

People can apply for benefits via the state’s COMPASS website, which stands for Commonwealth of Pennsylvania Access to Social Services. About two-thirds of benefits applications are now coming through COMPASS, according to state officials.

Because the application process is lengthy and complex, the state offers a much shorter “Do I Qualify?” eligibility screening tool intended to let people know if they will qualify for any of several different benefits.

Advertise with us

This is the tool researchers tested with hypothetical applicants to see if it would correctly predict eligibility.

The tool screens for eligibility for Medicaid, SNAP, cash assistance, free or reduced-price school meals for children, and Child Care Works, a subsidy that helps to cover the costs of child care for low-income working families. It was this that researchers say presented the most wrong answers to potential applicants.

“While these tools are designed to help struggling, overwhelmed people decide whether to embark on the lengthy benefits application process, their inaccurate eligibility predictions may prevent deserving households from seeking benefits,” researchers Nel Escher and Nikola Banovic wrote in their paper.

State officials say the tool is not an official determination of someone’s eligibility. The officials encouraged anyone who is uncertain whether they qualify to apply for benefits “so they can work through the application with a caseworker who will be able to fully review income, resources, and circumstances to assess what services or programs may be available.”

Ms. Escher said the issues raised by her paper speak to the broader problems with automating certain government processes. Often, as in this case, it is difficult to know if a system is functioning correctly without extensive testing, and there is a divide between those who built the technology and those who use it.

Advertise with us

Someone who is told they won’t qualify for benefits — even if they are told this incorrectly — is much less likely to go through the lengthy application process, the researchers pointed out. And even if the applicant does so, and learns the initial assessment was incorrect, they have no way of knowing the error extends beyond their individual case.

“The nature of how these things work is obscure. It does take the work that that paper did to identify that it is happening at a large scale,” said Sophie Song, national organizer at Movement Alliance Project, who is focused on inequality in algorithms and injustices in automated systems.

For this type of technology, there should be regular and transparent auditing, said Aaron Rieke, managing director at Upturn, an organization that advocates for equity in the design and use of technology.

“The kinds of work [the researchers] did here should be standard, and performed by the state. It’s almost always the case that problems are discovered after people start losing benefits, which is not the way things should go,” he said.

Families can apply for benefits online at www.compass.state.pa.us or drop off a paper application in a drop box outside their local county assistance office. They can call the Pennsylvania Department of Human Services hotline at 1-877-395-8930 to have a paper application mailed to them.

Kate Giammarise: kgiammarise@post-gazette.com or 412-263-3909.