How's Your BS Detector?

Carl Bergstrom, UW professor of biology, was frustrated. Everywhere he looked, he saw information presented as fact when it was actually misleading, biased, or based on conjecture. In other words, bullshit. And he saw people believing it.

So when Jevin West, associate professor in the UW Information School, mentioned plans to teach a course on big data, Bergstrom had a strong response. “I told Jevin, ‘That’s too bad. Because if you teach a course on big data, I’ll have to start teaching a course calling bullshit on big data.’”

That got a laugh from West, who said it sounded like a fun course. “I’d teach it with you,” West said.

Those five words led to a wildly popular UW course taught by Bergstrom and West, “Calling Bullshit: Data Reasoning in a Digital World,” and a book with a similar name.

But suggesting all that happened without a hitch would be, well, bullshit.

After their conversation, Bergstrom and West brainstormed how they would teach a course on calling bullshit and what content they would include. They focused on all the ways people misuse data to make claims that may or may not be true. They had no problem coming up with topics and examples, but they hit a roadblock when they proposed co-teaching the course. With faculty appointments in separate colleges — Bergstrom in the College of Arts & Sciences, West in the Information School — they discovered there was no mechanism to approve a cross-college course.

Abandoning the idea, they instead posted their proposed syllabus and other course materials online for anyone interested in the topic. They expected a handful of friends and colleagues to take a look. Instead, their imagined course went viral overnight thanks to a shoutout from an influential blogger.

“There were emails and phone calls and then stories in all the big US and European newspapers about this course that actually didn’t exist,” Bergstrom recalls. “People were excited. It obviously hit a nerve.”

Bergstrom and West proposed the course to the University again, as a one-credit pilot. It was offered in spring 2017 in a lecture hall with 180 seats. Registration opened at midnight; the course filled in one minute. “The people in the records office told us it was the fastest they’d ever seen any course fill,” Bergstrom says. “The demand was there.”

After the success of the pilot, a four-credit course, “BIOL 270/INFO 270: Data Reasoning in a Digital World,” was approved as a permanent offering. Though “Calling Bullshit” was removed from the official title, it is still used informally. Bergstrom and West offer the course annually.

Topics covered include spotting bullshit, the spread of bullshit, causality, statistical trickery, misleading data visualization, publication bias, and more. The main focus is the credibility of research results reported in scientific journals and popular media, though the concepts covered in class can translate to other areas where truth is compromised.

A central goal is to challenge the idea that those who collect and analyze data know more than the rest of us and their findings shouldn’t be questioned.

“The data in a research study have been constructed in a deliberate way,” Bergstrom says. “The person with the data gets to frame the story and the person without the data does not. When someone shows you data, they’re not beyond reproach. You can question their interpretation of the data.”

Of course, people lacking expertise in statistical analysis — in other words, most of us — may feel unqualified to question data. But the professors believe that the intimidating data-crunching phase of a study is rarely where the most troubling misinformation happens. More likely causes: how the research question is framed, how subjects are chosen, how data are collected, and how results are presented.

One example shared in class is a study showing that rap stars have half the life expectancy of country music stars. Digging into the study, students look at the source of the data — the age at which deceased country and rap musicians passed away. Rap music got its start in the 1970s, so few rap stars would have the opportunity to reach old age yet, unlike their country music counterparts. A clear case of selection bias. 

Another assignment has students create data graphics about single-payer healthcare, using worldwide health data. One group is told to argue for single-payer healthcare; another group argues against it. A third group presents what they believe is the objective truth. “The groups trying to tell biased stories learn how easy it is, and that makes them more skeptical,” says Bergstrom. “The group trying to tell the objective truth learns how hard it is to pinpoint objective truth in a complex dataset.”

West and Bergstrom stress that identifying bullshit is not enough. It’s important to speak out. They lead by example, regularly questioning the work of other researchers when they see flaws, including unintentional misinformation. Students get to see those interactions play out in real time.

“It’s fun to come to class and show them a story published that morning, then show them a tweet I wrote explaining why the story is wrong, and then the resulting retraction of the story that afternoon,” says Bergstrom. “They won’t get more timely evidence of the real-world impact of calling bullshit than that. They see how, by spotting that something is bullshit, you can change it. Not always, but sometimes you can. That’s an important lesson.”

Bergstrom is quick to point out that calling bullshit is not about putting other people down to make yourself feel smarter. “That just makes you a jerk,” he says. Instead, the goal is make us all smarter. “Providing objective, evidence-based criticism can be a very constructive, positive thing to do and a real gift,” he says.

Since creating the course, Bergstrom and West have become more involved in research in the field of misinformation. West is a founder and the inaugural director of the University’s Center for an Informed Public, which aims to resist strategic misinformation, promote an informed society, and strengthen democratic discourse. In 2020, he and Bergstrom published “Calling Bullshit: The Art of Skepticism in a Data-Driven World” as a way to share information from their course with a wider audience. The book has sold about 100,000 copies and has been translated into more than 20 languages.

“It’s a delight,” Bergstrom says of this turn of events. “What’s come from this course is more than we ever hoped or dreamed or imagined. Jevin and I feel super lucky about all that’s happened. And we hope more and more people will feel empowered to push back against misleading stories."