This transcript is auto generated and unedited.
>> Howard Bauchner: Hello, and welcome to Conversations with Dr. Bauchner. Once again, it is Howard Bauchner, Editor-in-Chief of JAMA. This is going to be a different type of conversation. It's going to be less medical, and I think interesting in a different way, at least I hope so. I am joined by Alexis Madrigal, an American journalist to spend of The Atlantic for about 10 years and Erin Kissane, an editor and content strategist whose love for the web runs deep. Welcome to both of you.
>> Alexis Madrigal: Hey, thank you, Howard.
>> Howard Bauchner: So, we're going to discuss your website, and it really feels like The Atlantic has upped its game around COVID-19. So, I'll start with you, Alexis. How did you conceive of the website? What is it that you wanted to accomplish?
>> Alexis Madrigal: Well, in the early days of the pandemic, we knew that there were very few confirmed cases, but there wasn't very good data coming out of the federal government or, initially, from the states either about how many people have been tested, and we now know that, obviously, there were many, many, many more infections than we were able to confirm with tests, and then the early days of the pandemic, a co-author of mind Robinson Meyer, we basically said, well, what if we just went to state websites and compiled everything they have? You know, like how many tests they done? How many cases do they have? Like maybe that would tell us something about the nation. And, you know, as we got launched into March, we'd only tested say a couple thousand people. So, our tiny case numbers were a reflection of the lack of testing not a lack of infections, and that effort turned into what became the COVID Tracking Project. Erin came on. There was another guy named Jeff Hammerbacher, who's a data scientist and does bioinformatics, and from there, we just were in the response in a lot of ways. It was really an emergency response kind of operation. We didn't actually expect to still be doing this all these months later. We thought that within a few weeks, all the data that people would want and need in order to understand what was going on with the pandemic would be available, and it has just turned out not to be that way.
>> Howard Bauchner: Erin, how did you get pulled into it, and what your background?
>> Erin Kissane: Sure, I actually -- Alexis mentioned Robinson Meyer, and I've been friends with Robinson for a long time, and Rob is one of the only people back in late January/early February that I do myself who was sort of also sitting up late at night reading news coming out of China about the pandemic there and what it looked like was going to happen and some late night texts right around when he and Alexis decided to stay up all night and compile all the data for the first time. So, as soon as they opened the door to the volunteer support for that work, two people, it turns out, can't actually do that and sleep. Apparently, I was, literally, the first volunteer in the door, but I'd also been watching it, sort of, as it moved into the public view, and I'm an editor and a community manager and sometimes events person who's worked in journalism and tech. So, most of my work in the last 10 years has had something to do with open data and with data journalism. So, there were some obvious connections. But again, it certainly was not how I had expected my year to look, even when I had joined on with the project. As Alexis said, this was something we thought would be quite temporary.
>> Howard Bauchner: Yes, there was a great story yesterday. I guess of family was trying to get from Georgia to Fairbanks, Alaska, to join a woman and her children, to join her husband was in the military, and they got to Canada, and she couldn't keep driving because it was winter. And someone helped her drive the final thousand miles. And he just said he had been in the Canadian military, and he said, it's just an unusual year. You just have to help people. It was, you know, so basic and stark. How many people are involved in the project? Full-time people? Volunteers? Because the data's up every morning. I get up around three or four the morning. I immediately go. I go, whoa, it's updated yet again. How many people are actually involved in the project?
>> Alexis Madrigal: Yeah, I think, overall, it's probably been something, at this point, close to a thousand people have contributed in some way. There's 30 staffers of one type or another, and there's about, you know, 300, 350 active volunteers. You know, that core data run that you see every morning is executed by a whole team of people. There are, you know, people who go to the state websites who are sort of the checkers. There's double checkers. There's ship leads. There's a data quality team that's monitoring all the things that are happening. There's little automated components. I mean, there's now an entire sort of process and system, because as Erin noted earlier, I mean, it's easy enough to go to every state website once and write it into an Excel spreadsheet. Very difficult to go it over and over and over again, particularly as state data definitions have really shifted. So, a ton of our work now is trying to understand when one state reports in case, what do they mean, versus when another state report the case, or a test or a hospitalization?
>> Howard Bauchner: Erin, what's your -- has your role changed over the last five, seven months? Do you do something different, and are you checker? You know, do you oversee people? I'm trying to figure it out. It reminds me a little bit of how I keep reading how Wikipedia runs. But I'm curious what your current task is.
>> Erin Kissane: Yeah, I signed on to do data entry, and then wound up coming in as what we decided to call managing editor.
>> Howard Bauchner: Oh, okay.
>> Erin Kissane: You know, Alexis and I come from one kind of journalism or another. So, that was a familiar thing. We say that we -- certainly, a lot of the work is based in sort of a shared brain, at this point, but I focus more on the inside of the house. And so, we have that data shift that does, you know, the testing and outcomes data that we do daily. We also have a whole team that collects data about cases and deaths in nursing homes and long-term care facilities. We have another team that collects data that's specifically about racial and ethnic demographics, and then, we have a whole web light organization. We've got an editorial team. We've got a lot of teams. And so, a lot of my work is running our editorial side with a few other people, but also helping make sure that all the sort of synapses are getting connected across the teams, so that the work that's happening in data entry and moving through the data quality researchers also flows into, you know, our communications out to states. We've had to build sort of apparatus for, you know, communicating with various parts of state and public health departments, state officials, state electives, and so on. So, keeping the pipes connected internally and externally is a lot of what I do. We also do a lot more editorial work, as in, you know, communication. We had a science communication lead now, Jessica Malaty Rivera, who is incredible and has brought a lot of public health expertise. But we wind up doing a lot more explaining on the data than we certainly expected to when we were, literally, people making a spreadsheet. So, there's this whole sort of universe of things that we do around the data now, and that's just grown up more like a coral reef than a planned organization, because it's been as needed. So, I do what's needed.
>> Howard Bauchner: Now you mentioned quality. You know, there's the Hopkins side, the New York Times side. Mike Osterholm has his side in Minnesota. I tend to go different ones when I'm worried. I was going to Maine over the summer. So, then I go to the Maine site just to see what looks like, and I was going to travel this weekend, but the number of hospitalizations has made me nervous. So, I'm actually not going to travel this weekend, which is unusual for me, because I do try to get back to Boston, and you mentioned quality. Are you accurate? And I don't mean that in an offensive way, but, and your numbers compare to other sites? The subjects are often a little off in the CDC. Not just yours, but all of the other ones. Do you have a sense if you're accurate, and how do you know that? Either of you can answer that. That's fine.
>> Alexis Madrigal: Yeah, I'll hand it over to Erin in a second, but I think there's a few different ways to answer this. I mean, are we accurate and that we put the right number in the box that's on the state website? At this point, the system for that is very solid.
>> Howard Bauchner: Okay.
>> Alexis Madrigal: There's a deeper question, though, about where -- do these numbers all mean the same thing, right? This is a patchwork data set, and the court attention of the project has always been that we're both provisioners of data, we put it out there. People use it to do all kinds of things including the New York Times and Hopkins and other places. But we're also critics of the data. We point out the ways that our pipelines have failed us and that numbers that may look the same for different states may actually have a different meaning. And so, I tend think that that is actually very productive tension, because it provides something I've heard over and over on your show is people trying to be humble about what it is they know that's happening in the pandemic, and I think these numbers, particularly state-compiled numbers, should make us humble. I really do, because, you know, you mentioned the differences between deaths. I mean, as our researchers have gone deeper and deeper into the ways that states do death counting, we just keep finding a lot of ways in which states and the feds and other people are really doing things in different ways but calling it the same thing, and it's not that it calls the overall trajectory and arc of the pandemic into question. It totally does not. But on the specifics and in the details, it really does matter.
>> Howard Bauchner: What you're saying, I just wanted to comment on the counting of deaths. It is interesting. Steve Wolf has written for us a few times, and Steve is one of the country's experts on this, and he and I have talked about it. And so, the metric we have really pushed hard on is excess deaths. We know some are attributable directly to COVID-19, but then there's the number of deaths every year is something the CDC counts well, actually. And sadly, a death is a yes or a no, and we really know how many deaths do we have month-to-month this year versus last year, and then there's a certain number that are related to COVID-19. So, we know deaths from motor vehicle accidents have gone down, but we also know that deaths from myocardial infarction and stroke have gone up. So, Steve has been really masterful about talking about excess deaths, rather than trying to say these are the number of COVID-related deaths and then all the other deaths. Back to you about quality, though, Erin, because I'm interested in your take on the quality issue.
>> Erin Kissane: Absolutely. You know, accuracy is something that concerns us so much. We try to be as precise as we can in our reflection of what states report, but what I tell reporters when I'm on the phone trying to explain this data is that, you know, the differences you see across different compilers, across dashboards of various kinds that aggregate this kind of data, they're pulling from different parts of the pipeline. If you go to the county level you use scrapers and you update multiple times a day versus if you do manual collection in the way that we do it and you go to the state level, everything is going to be just slightly off in terms mostly of timing. So, I think it's certainly true that to us, as a project internally, we're tremendously concerned with these definitional differences and trying to learn as much as we can about exactly what a state means when it puts, you know, COVID-19 hospitalizations. Are these confirmed, or confirmed plus suspected, and so on and so on, but in general, I want to say, I think there are many correct, accurate ways to aggregate this data, and we don't really see that, you know, we think that ours is accurate in ways that The Times is not or USA Facts, and so on. But the other thing is, you know, I think, all of these metrics are coming from pipelines themselves that move differently. Cases move on one cadence. Deaths go through a different reporting process. Tests, especially negative tests, you know, that's been a huge problem for states because they haven't had to report negative tests in the same way for other diseases. So, it's an overwhelming across-the-board. Each of these metrics moves differently three time. And so, it's very difficult to match up and say, you know, we're seeing, on this one day, the cases reported maybe from one period of tests reported, maybe, from a different period, and deaths and so on. So, we try to do education about the fact that these numbers are real numbers reflecting the real experiences of human beings who have been sick, who have been hospitalized, who have died, but they are not necessarily synchronized in a way that allows us to make really precise comparisons, especially for things like tests positivity. It gets really tricky because you have tests in cases moving at different paces through slightly different definitions across jurisdictions. So, it's a yes-end answer. We think that we are accurately reflecting what states report, but we try to persuade our contacts in different news organizations to take a lot of care with house a describe what the numbers mean.
>> Howard Bauchner: Yeah, I think the trend data, you always are showing graphs. The trend data are really very helpful, and then, obviously, a lot of the seven-day averages. And then, when you give state data, what I like is you just give the state the absolute number, but then you often give a per thousand, per 10,000, per 100,000. Because I can't really understand California compared to Idaho without a denominator So, you do it per population, and if I can just click, I can find it. Have you wanted call up a state? I'll pick North Dakota, and say, do you know you're just doing it wrong? Could you just change the way you did it? Have you wanted to call a state up and go, do you really know you're an outlier?
>> Alexis Madrigal: Yes, go ahead.
>> Erin Kissane: I can take that one.
>> Alexis Madrigal: We have a lot of contact with states.
>> Howard Bauchner: I'm sure you do.
>> Alexis Madrigal: Here's the first thing I want to say. Those people are heroes. Like that is such hard work under incredibly, incredibly difficult circumstances. You're working around the clock and they have been since the beginning of the pandemic, and like the very first and most obvious and most heartfelt thing to say is like just total respect for the difficulty of the task and how they risen to it. Two is, you know, the states have tended to want to reflect their internal needs, and their internal needs means the governor wants this. There state epidemiologist wants that. When they think about the numbers they put out, it's really about their state. And so, we've never wanted to say, hey, you should report this stuff that your state wants to report. We've said, hey, maybe you can also provide this other data set. Because then, North Dakota and South Dakota will make sense if you look at them together. Rhode Island and Massachusetts will make sense if you look at them together, and what we've found, over and over, is that when we make the case like that, like hey, we totally understand the reasons why you're doing what you're doing, and also, we can have a better national dataset, if you're willing to release these numbers, and that's been our approach throughout is don't release things to us. The COVID Tracking Project at The Atlantic does not want to be the holder of data. We want you to release it to the public, and we want to understand what's in those numbers, and then we will compile it in a way that makes it comparable with the states around you and the rest of the nation to the extent that we can.
>> Howard Bauchner: Erin, your sense about this, you know, reaching out or working with states, is it similar?
>> Erin Kissane: Yeah, I just want to say, it's very important for us to stay in our lane. We try to do the best job we can using public data to stitch together aggregates that are at least useful summaries, that are at least useful directionally, but we aren't in a position to say to a state, you know, you're doing this the wrong way. What we often wind up doing with a status saying can you explain to us exactly how this works? How are you counting repeat positives in this metric? Are you, you know, are you following the CST definition of April as opposed to August, so that we can at least annotate and show why your definitions may be different rather than coming in and saying, you know, you're doing it wrong? But something that's really, I think, been an important part of our work at various points throughout the pandemic is to be able to tell members of the public and media, you know, something we haven't seen. We see this tremendous variance, and we do see inconsistencies in the way states report in cadence and definition, and so on. What we do not tend to see is evidence of manipulation, of bad faith. We see a lot of people working tremendously hard, and furthermore, we think it would be really difficult to have a state work up a conspiracy such that every county public health department is, you know, is going to go in on this large-scale data manipulation. That's something we don't have any evidence of. So, we really think that there is every reason to trust that every state is doing the best it can, and also, I mean, the states have really, in many cases, unequal resources for what they actually can do physically with the humans and systems they have. Most have improved tremendously over time, but what we don't see is, you know, nefarious activity in the data. So, that's been really important for us to try to, you know, help people understand why they should trust this data. And sometimes, that's even related to other data sets. We've been very recently able to work with some federal data sets that turn out to match very closely what we compile from states, which allows us to say, this is probably trustworthy data. We can, you know, if we adjust for these definitional differences and move, you know, this data set's one day ahead of the other one, we get a match within 2%. So, probably the HHS hospital data is pretty good. You can probably put your faith in that. So, that's part of the work that we feel like we need to be doing.
>> Howard Bauchner: Your comments are just incredibly reassuring. You know, not to think that there's someone sitting in some state, and some office, how can I manipulate this in some way to benefit who knows what? But that the vast majority of people working at the state level are honest people who simply want to report out the data, so that people can try to use it in a constructive way. That's very reassuring. Now both of you must get emails, because it wasn't very hard to find you, and say can you report this? Or could you do that? What's been the biggest call for more information that you haven't been able to do? You do so much, but what do people ask that you just haven't been able to pull off?
>> Alexis Madrigal: Yeah, I would say that it's been going down the geographic scale. It's very difficult to find the set of numbers that we track at the county level and to standardize for them. I mean, this is another reason why we're very excited about some of the stuff that's coming out of HHS, because, you know, they're now beginning to release facility-level hospital data which is just so much further down closer to reality than the state aggregations. Any other things, Erin?
>> Erin Kissane: Well, also, there's demographic data that's been impossible for us. Age brackets are the thing that's been one of the most frustrating things, because obviously, with this pandemic, you want to be able to break things down by age, but when you have 56 jurisdictions of states and territories, and they're all using incompatible age buckets, there's no way. You know, we've tried a bunch of ways to get into that, but that's something that is going to have to come from, you know, the line-level data at the federal level, and we are seeing more of that, but it's not something that we have been able to stitch together to the tremendous frustration of many members of our team.
>> Howard Bauchner: When I compare, you know, sites or articles that we get, people do 15 to 25, 26 to 35. But then, someone else will do 15 to 35, and I go, look, I'm trying to compare, you know, data sets. So, I know the feeling, and you know, I look closely because of the current deliberation about the two vaccines that are likely to win EUA approval, and very effective, but if you look at the granular data about immunogenicity in older people, it's not nearly as good as the media's often portraying it, because they've focused on a single number. The confidence intervals are much wider, simply because the numbers are smaller. But it is quite possible about 80% of the deaths are in individuals older than 65, and it remains somewhat unclear how successful that vaccine is in that age range. But sometimes, they cut data greater than 60 or greater than 70. So, it's hard to grab that data. Alexis, I don't even know who owns The Atlantic. I assume it's for profit, or if it's publicly traded. It's never been known as a magazine or a website focused on health. Who made --
>> Alexis Madrigal: Yeah, how did that happen?
>> Howard Bauchner: Who made the grand decision? I'm sure this is costing money.
>> Alexis Madrigal: Well, yeah. So, here's how it works, basically. We got this thing running as a volunteer group thinking we'd only be doing it for a little while. The Atlantic said, with my time, said go ahead. Run out there. And then, as it became clearer that we were going to need to actually develop a real organization and institution, we went back and we said we'd like to raise some money from philanthropic sources. And so, The Atlantic, basically, set up some structures for us to be able to bring in money with them taking, actually, a much smaller overhead than a university would, about 12%. And then, the rest of the money goes to pay people. So, why did they agree to do this? It's an interesting question. I mean, any part of it is we were rolling. I mean, you remember back to March and April. We had, I mean, we have the same people we had before, actually, right, in a lot of cases. But it was just so clear that this is a necessary component of the information ecosystem, and it was also something that couldn't be done inside The Atlantic proper. I mean, we have as many staffers as like run the website. You know, so it's really quite a big operation, and I think, honestly, you know, when I met with Jeff Goldberg, our editor-in-chief, I mean, I said listen, we're producing this public good, and we can't stop producing this public good even if it's kind of an unusual and even uncomfortable position for us as journalists to be in this space. And he was like, "I agree. It is uncomfortable, and you can't stop producing it. So, go raise the money." And we're now supported by the Rockefeller Foundation, McGovern, Chan-Zuckerberg, Emerson Collective who are also the majority owners of The Atlantic, and it's been, you know, kind of an amazing thing to be able to take all these people who came in as volunteers and just because they wanted to help and basically make it, you know, something like a job for a lot of them, and, you know, some of our people, this is the thing that they want to do with their life I think henceforth. I mean, I think you're going to have an entire CTP cohort of health data people who go out into the world because they've discovered this is really their passion.
>> Howard Bauchner: It's been interesting to see, you know, which groups have really taken up the task of trying to help people understand what's happening. I'm very fortunate that JAMA is published by the American Medical Association, and we made a decision early in the pandemic, as we've done with every pandemic, although this one has been larger and more substantial, with far more commitment of resources. All of our COVID-19-related content is free to the world, as we've always made that commitment in the middle of a pandemic, but as I said, this has been far more all-consuming, and it's not just JAMA. It's true for all of the major journals.
>> Alexis Madrigal: We really appreciate that, by the way.
>> Howard Bauchner: You've lived the data. You see the data every day. I'll start with you, Erin. What does it mean to you? From a scientific and personal and an emotional standpoint if you step back and you look at the numbers, what does it mean to you?
>> Erin Kissane: I think that an important piece of the COVID tracking project internal culture that has emerged from this group of people who really do come from a tremendous variety of backgrounds. So, I'm kind of speaking for the we here instead of just me is that we come into these numbers every day, and we increment, we'll say deaths, because that's what a lot of it comes down to. We increment these numbers up or down across all of these jurisdictions, but it remains, I think, central to all of us that each of these numbers is a life. It's someone who got sick. It's someone who got really sick and is in the hospital and is scared and can't see the families. Or its someone who we've lost and whose families and communities have sustained a loss. And so, you know, you might think after 10 months of doing the work that we would be at least prepared to face things like big jumps in numbers of hospitalizations or big jumps and deaths or cases, because we know that three weeks after cases, we're going to see it in deaths. But I, for me personally, it's a hit every time, and I think it should be. I think if we didn't feel that, we would not be doing the work with the kind of care that it deserves, and I know that I speak for this whole apparatus of people who are volunteers or staff who are, many of them, literally, every single day facing these things and don't take days off. You know, it's quite affecting for all of us, and I think if it weren't, it would be a problem. I think it should be affecting for all of us, because that makes us look that much harder at, you know, how can we do the best job we can of bringing this information, not just into the public view but explaining what it means and trying to help journalists and other communicators use it to try to save as many lives and prevent as many infections as they can?
>> Howard Bauchner: Same question to you, Alexis. What does it mean, you know, when you step back and you see these numbers? Have you digest it?
>> Alexis Madrigal: I think that the process of working with these numbers, and Erin, I really feel like, is like the soul of our operation. So, I'm borrowing her formulation here, but it's kind of a devotional task. You know, I mean this thing is it's impossible to do the task that we're doing. Like you can't actually standardize a national data set. You know what I mean? I mean, it's really difficult and emotionally tough work, and everybody knows that who's working on the project, and yet, we come to it each day as part of, I think, what for all of us is our own personal process of kind of a form of collective grieving within our team. You know, it doesn't look that way always in our slack and in our communications. We have a very like serious culture of gratitude and a lot of, you know, a humor and love that passes between people on the project, but I think part of it is is that the fundamental thing is this kind of record keeping of the failure, record keeping of death of counting all these lives, and it's really hard for us. I mean, honestly, I think the only thing that really keeps a lot of the people going is that we're bonded to each other through doing this work, and that is how people are able to just, you know, face it day after day.
>> Erin Kissane: But also, I just -- I cosign all of that, but we also, I think, all of us are very, very aware that like, yes, this is taxing work. It's cognitively taxing. It's emotionally difficult, and yet, we are doing it in service of frontline workers who are risking everything they have. So, you know, our work is as much for healthcare workers as it is for people who are, themselves, trying to avoid, you know, getting COVID or trying to keep their families from getting together. It's for those people who are going through something that is vastly more difficult than anything we do. So, just to keep it in perspective.
>> Alexis Madrigal: Yeah, totally.
>> Howard Bauchner: Yeah, it's been interesting the last couple weeks have been so difficult, this balance, the optimism about a vaccine versus the numbers that you have produced. That, you know, 100,000 people in the hospital, record number of deaths, no end in sight. I just spoke to Steve Wolf earlier today. We think excess deaths compared to last year will be above 500,000, which will make it more deaths than World War II. Last year, there were about 2.8 million deaths in the US. I think Steve thinks there could very well be 3.3 or 3.4 million by the end of the year. And I think what's been difficult for many people, and I'm sure it's been true for the two of you to try to convince people just a mask and socially distance. I think that's been painful. I appreciate the politicization of it, but, you know, I'm hoping around the new year that religious leaders will stand up and say, you know, the basis for so many religions is love your neighbor, care for your neighbor. Well, part of that now is to try to keep them safe and healthy. And you know, I am a physician, and for the first time, I feel like healthcare workers are frustrated. You know, they've always just committed to taking care of people, but I think when they see people coming in who haven't masked, they're going, you know, you're really not helping me. You know, you're making my life much more difficult, and it's not necessary. I do hope that there's a change. I just don't know how we can get there.
>> Alexis Madrigal: I think that a core principle of ours has always been to try to be a force for reality, and I think that this kind of unites the questions here that we've been talking about. You know, our people want to be that force, and there's been so much misinformation coming out about the pandemic and about the numbers specifically, and it's never been in the service of actually developing an alternative view of like what's happening. It's always just been with, you know, reference to like climate science. It's always been kind of like that. Just trying to sort of create a cloud of uncertainty about what's really happening, usually for political ends. And I think it's had just a tremendously bad impact in the United States. And so, I do, I mean, when we wake up every morning and we go do this, I mean, some of it is if we can be that force for reality, if we can just keep pressing back on people who are not actually trying to see what's happening but at have a political axe to grind, then I think that's how we can help the most, and that's how we can help people on the front lines the most.
>> Howard Bauchner: This is Howard Bauchner, Editor-in-Chief of JAMA. I've been talking with Erin Kissane and Alexis Madrigal. I'm sorry if I mispronounced you last names. I'm terrible with names.
>> Alexis Madrigal: You're doing great.
>> Howard Bauchner: Everyone knows that by now, and I've been talking to them about a remarkable project, the COVID-19 Tracking Project that's been up and running for many, many months by The Atlantic, and I want to thank the two of you, all the volunteers, and all the forces at The Atlantic that have allowed you to do this. I look at many websites every day, and years is certainly among the best. It's an incredibly important source of information. So, thank you very much. Stay healthy and enjoy the holidays.
>> Alexis Madrigal: Thank you Howard.
>> Erin Kissane: Thank you for having us on.
>> Howard Bauchner: Buh-bye.