The numbers should speak for themselves, right? You hear that on occasion, especially in the field of visual journalism and infographics. But there’s a reason Mark Twain famously uttered the phrase, “Lies, damned lies and statistics.” It’s because numbers do not speak for themselves, and viewed apart from their proper context they can paint a skewed picture, or even one that does more harm than good.

In a recent graphic, I charted traffic fatalities broken down by categories, including drunk drivers, drivers without seat belts, etc. One of those categories was a statistic about older drivers, who die more often in car crashes than younger people. Aware of the stereotype, I worried about reinforcing a common prejudice that older drivers are more dangerous and prone to accidents than younger drivers. Without the proper context, the numbers could be interpreted that way. However, as the CDC states “Higher crash death rates among this age group are primarily due to increased vulnerability to injury in a crash.” In other words, it’s not their driving, and with the proper context, the numbers seem to belie that fact.

From the beginning of the COVID-19 crisis, my colleagues and I have been trying to understand what recovery from the disease means, how it is determined, by what standards someone is labeled as recovered and whether those standards differ depending on their source.

As a numbers-focused person whose job it is to understand and present stories as visual data, I was drawn to the Johns Hopkins COVID-19 data dashboard. It has since changed significantly, but early on, one of the data categories was “Active Cases,” which was defined as COVID-19 cases minus deaths, minus recovered. But what does “recovered” mean?

Advertise with us

It seemed to me that the definition of recovery really mattered for two reasons: as a consideration for when the economy would be allowed to start up again and because people who recover from viruses unwittingly acquire a superpower: They become immune to the virus. At least that’s what I originally thought. It turns out that recovery and immunity for novel coronaviruses are not so straightforward..

To find out how recovery was defined and how it was determined, I turned to one of the people behind the COVID-19 dashboard, Lauren Gardner, co-director of the Center for Systems Science and Engineering with Johns Hopkins University. In an email, I asked: What are the criteria that determine recovery? How are those cases confirmed?

Her response was, “We are simply reporting the recovered cases [that] are reported by location, which should be based on patients testing negative for the virus, and being thus released from healthcare.”

I carefully parsed her words and, not satisfied with her answer, I asked some follow-up questions: Does “should” mean it may not necessarily be the case? Are the “reports by location” vetted or accepted at face value? Does “being released from healthcare” stand alone as an indicator of “recovered” or is that paired with “testing negative”? I also secretly wondered if she was being deliberately evasive.

Given the harried state of things in those first weeks of COVID-19, I was not surprised that she didn’t reply to my second email. I tried asking the questions again through the website’s media inquiry form. The answer I received was pretty much the same, which read in part: “The Recovered category counts reports of individuals who have tested negative and were released from healthcare facilities.” There was nothing said about the vetting of those reports or their reliability. It felt like they were avoiding the real question.

Advertise with us

Call the doctor

It’s not often that I feel like I’ve hit a dead end, but when I do, I can turn to SciLine, an independent service that connects journalists with experts. Before the end of the day, I had a name and contact information for the expertise I needed: Donald Thea, professor of global health at Boston University.

Dr. Thea said that I was attempting to peek inside the “black box.” “I knew it!” I thought, but only for a second. He went on to explain that there are currently no WHO- or CDC-defined guidelines for defining recovery. It became clear that nailing down a coherent definition for recovery is a challenge partly because there are so many unknowns about this disease and the effects it has on the human body, short- and long-term. As Dr. Thea noted, “Given that there seems to be an increasing appreciation that some patients have a slow or long recovery with, at least temporarily residual debility, one needs to use that term [recovery] with care.”

There are reports of long-term damage from COVID-19 — widespread damage to organs including the lungs, heart, kidneys, liver and brain, for example — and it is possible that some of the effects of this virus, for some people, may never go away. Which prompts the question, are those people ever fully recovered?

Further complicating the topic of recovery is that there are no systems in place to monitor quality control for tests, which affects their reliability. There are problems that exist at the testing level. For example, Dr. Thea described seeing swab tests being done incorrectly. The swab is supposed to go into the nostril, straight to the back of the sinus, parallel to the roof of the mouth.

Advertise with us

Tests are usually developed and tested by the FDA under ideal conditions and field conditions. But because the deadly and timely nature of the pandemic doesn’t afford the luxury of the usual test development process, the medical community must continue using whatever tests are made available to them. Dr. Thea emphasized that while there are currently suboptimal and temporary solutions, they’ll get better in time.

With regard to immunity, Dr. Thea says the ideal case is one in which an individual who is exposed to a virus develops immunity to the virus. But we just don’t know enough yet to say whether that is true for COVID-19.

One conclusion I’ve drawn at this point is that recovery can be defined in at least two ways: (1) No longer infected or contagious and (2) free of symptoms or damage from the virus. The first implies the individual was tested positive for the disease and is now virus-free. However, that same person — although recovered in a strict, narrow, technical, no-longer-infected sense of the word — may yet experience long-term, even lifelong damage from the virus. That leads to a second, more broadly defined sense of the term, and by that definition, some might never fully recover.

Lack of clarity

There is part of me — probably the more analytical and statistical side — that feels manipulated by the lack of clarity and wants to expose the absence of a standard in recovery reporting, the unreliability of tests, the vagueness of definitions and overall inconsistencies, which could cast doubt on the whole enterprise. But, as my colleague, health and science writer David Templeton points out, thoughts of being manipulated or concerns about secrecy "may occur whenever there's a new disease no one understands.”

Advertise with us

Then there is my human (and thankfully more dominant) side that worries about casting that kind of doubt and doing more harm than good in the current climate. By focusing on the flaws and inconsistencies, would I discourage people who need to be tested from getting tested? Would I be providing fodder for skeptics and gainsayers to further dismiss the threat of the pandemic?

I see now that my focus on recovery and immunity was very premature. What might have appeared to be deliberate evasion or an intentional “black box” scenario was rather due to dealing with a new virus and not knowing very much about it.

Dr. Thea said, “I, frankly disregard [recovery data] wherever it is reported.”

So, what is the right thing to do, journalistically speaking? You might say, “You’re a journalist; show the data. Let the numbers speak for themselves.”

But without the proper context, it is all “lies, damned lies and statistics.” There are some who would say I’m overthinking this and not giving the reader enough credit. Others may say I’m searching for a solution to a problem that doesn’t actually exist. But it is something that keeps me awake at night and I’d rather wait for that context and more complete knowledge of this disease and what recovery looks like than risk doing more harm than good.

James Hilston is an editorial staff artist and makes informational graphics for the Post-Gazette. James Hilston: jhilston@post-gazette.com.