Mistrust and Verify
“There are no stupid questions, only fact checkers who regret not asking them.”—Author unknown
As an editor, I’ve always had to do a certain amount of fact-checking—names, dates, places, and other facts. I thought I was fairly good at it, since I’d been fact-checking in one form or another for 30 years. Before the Internet came along, I had a massive set of the Encyclopaedia Britannica that I used to dig through, sitting on the floor surrounded by open volumes. And since the Internet, I’ve spent a lot of time online doing research, surrounded by open tabs on my browser.
In 2014, however, I got a project to fact-check an entire book, and it became clear to me that I had a lot to learn when it came to fact-checking and online research. I finished the book and the client was happy with the results, but I wasn’t happy with how long it took me and how frustrating it was to locate and verify some of the facts (or not locate and verify them in a few cases).
Since that project, I’ve been studying fact-checking, when I’ve had the time. I shared what I’d learned in a series of blog posts between November 2015 and July 2017. Now I’ve gone through those posts, reorganized them, and compiled the best points into a PDF, in case the links and advice might be helpful to others.
The topics covered in this blog (and the PDF) include books and news articles, blogs and resources, links, search techniques, and sometimes things that just strike me as interesting. I’ve also included things that may seem pretty basic to some readers, but little things can make a big difference if you’re not aware of them.
This isn’t a blog about political fact-checking. I’m an editor, not a journalist or a campaign worker. It's verification.
I’m calling this “Mistrust and Verify” in honor of a friend I used to work with. When he’d proofread something, he’d say, “There’s a mistake on this page, and I’m going to find it.” And since he was a good proofreader, he often found quite a lot. That’s a helpful attitude to have in fact-checking, too—not “These facts are probably correct,” but “These facts may well be wrong, and they’re not to be trusted until they’re verified.” I don’t recommend skepticism as a personality trait, but it can be very helpful in fact-checking. As Craig Silverman put it in his Verification Handbook, the basic principle of verification is “How do you know that?” And, if you’re going to be thorough about it, “How else do you know that?”
So here’s some advice on “How do you know that?” Or at least on how you can look into it. You can read or download my PDF of past blog posts at the bottom of the page. It's been revised as of July 2019.
February 2021: “Fact-Checker, Fact-Checker, Check Me a Fact”: A good overview of the fact-checking process by Sasha Nyary, via the ACES newsletter.
January 2021: From Loránd Bodó: “Constructing powerful search queries in OSINT investigations.” Good examples of complex queries, along with a sample chart he used to keep track of his keywords. (Link thanks to GIJN.)
A new series from Dan Russell at SearchReSearch that looks to be very helpful. The first installment: “How to Find ... anything. #1: How to find Do-it-yourself information”
November 2020: From SearchReSearch: Looking for data? Here are 3 Google data set sources. Note that the third source requires a bit of coding in Google Sheets.
October 2020: A post from Lifehacker on “Where to Find Public Records Online” with some good links
I finally finished reading The Joy of Search, by Daniel Russell. Great book for anyone interested in online research, and simply an interesting book, period. If you’ve ever been curious about anything and decided to look it up, you’ll find this book helpful.
Good advice on fact-checking (with good examples) in a well-written research paper (a 56-page PDF): Sam Wineburg and Sarah McGrew, “Lateral Reading: Reading Less and Learning More When Evaluating Digital Information,” Stanford History Education Group Working Paper No. 2017-A1, October 6, 2017; http://dx.doi.org/10.2139/ssrn.3048994
September 2020: Dan Russell addressing the question “Your everyday fact-checking? What do you do?” with some fact-checking principles and examples.
July 2020: Dan Russell of SearchReSearch with tips on “How to find collections of online content”
“Six Tips to Fact-Checking as a Copyeditor,” by Erin Brenner. Good advice. As the post says, “The real trick is thinking about just how wrong the facts can be and working to prevent that.” When fact-checking, you won’t go astray if you assume the facts are in error unless proved otherwise.
May 2020: I've fallen way behind on this. From SearchReSearch: Finding extramusical sounds that people make during performances. A good reminder of how to do online research. Well, all Dan Russell’s posts are good reminders and have helpful information. I just happened to enjoy this one.
December 2019: “The Rise and Fall of Facts”: A look into the history of fact-checking by Colin Dickey in the Columbia Journalism Review.
“Catching Potentially Expensive Errors of Fact”: From ACES: How a simple fact-check by an editor saved a company time and trouble.
November 2019: Some basic search tips from Daniel Russell: “6 steps to being a smart searcher”
An interesting and useful Verification Toolset, mentioned at the 11th Global Investigative Journalism Conference (2019).
September 2019: An interesting post from ResearchBuzz on Google News and its glitches: “Google News Is Driving Me Bonkers”
An interesting article from the New York Times on the lack of fact-checking in nonfiction books: “It’s a Fact: Mistakes Are Embarrassing the Publishing Industry” (paywall).
August 2019: Some worthwhile questions to ask yourself before starting a research project, from Mary Ellen Bates
A new SearchReSearch post by Dan Russell discusses the use of the Google operator “after:” as in [ mapping resources after:1-1-2016 ] for example, to find recent resources. He also mentions the value of following experts or enthusiasts in the field you’re researching, which is another helpful tip.
June 2019: The Tow Center for Digital Journalism just posted (June 7, 2019) “A Guide to Open Source Intelligence (OSINT),” which is helpful. Note that the "Tools" section in the links is actually at the end of Part 7 in those same links.
May 2019: From the Columbia Journalism Review: The perils of publishing without a fact-checking net
This article in NY Mag has provoked a lot of discussion among editors: “Here’s an Actual Nightmare: Naomi Wolf Learning On-Air That Her Book Is Wrong.” A lot of the discussion has been about why book publishers don’t have fact-checking departments, and some has been about the lack of scholarship that led to the error. My reaction, as someone who fact-checks books and news scripts, is “Wow, God help me never to let something like that happen to a client.” I’ve been rethinking my work processes to be sure errors like this won’t slip through.
From “Michael Wolff is back,” by Jon Allsop in The Media Today newsletter (May 31) from the Columbia Journalism Review: Early last year, Michael Wolff’s book Fire and Fury—a rollicking, fly-on-the-wall account of dysfunction in Trumpworld—devoured so much press anticipation, and picked up so many pre-sales, that its publisher released it four days ahead of schedule. Wolff’s next book, Siege: Trump Under Fire, is due out in four days, and the reception has so far been different: less buzzy, more reflective of factual errors and uncorroborated claims. … In an interview with Michael M. Grynbaum, of the Times. Grynbaum asked Wolff what efforts he had made to corroborate and fact-check his new book. … “I actually don’t believe, if you know the answer, it is necessary to go through the motions of getting an answer that you are absolutely certain of,” Wolff told Grynbaum. [Frank: Not taking sides in politics here, just expressing some horror at that last statement.]
April 2018: Some interesting posts on research and fact-checking that I’ve come across:
The Wayback Machine Browser Extension
Daniel Russell mentioned this in a post from March 9: If you do active online research, you fairly often run into web pages that are 404. When this happens, it means that the link you're trying to follow leads to a page that is missing. You can use the Google cache operator to see the cached version of the page: cache:my-broken-url.html. And if that doesn’t work, try the browser extension for the Internet Archive’s Wayback Machine, available in Chrome, Firefox, and Safari. I have the Chrome version installed, and it’s been useful.
Ren LaForme at Poynter.org has a weekly email newsletter called “Try This” featuring tips on technology for journalists, and they often contain helpful information for fact-checkers and researchers. The March 12 newsletter says that “Discourse Media, a Canadian independent news company, just released a spreadsheet full of databases for research and fact-checking. For each, Discourse notes what type of information is included, the region it’s useful for, handy tips and whether an account is required.”
On April 2: Happy International Fact-Checking Day! You should celebrate by learning from one of the best online investigation outlets in the world. Bellingcat has used tools and technology to file deep reports about big issues like Russia’s incursion into Ukraine and the Syrian civil war. And we have a cheat sheet featuring some of their most-used tools.
Research Using Google Sheets
An interesting ResearchBuzz post by Tara Calashain on “Using a Google Sheet to Build Search Queries for Unknown Topics.” Looks like a very helpful tool:
Last week I got an e-mail from a reader named Joe asking for help researching a topic he didn’t know a lot about. Then at about the same time, I read about an add-on that would let you import Wikipedia data into Google Sheets. And I thought, “Hey, this might be a nifty way to quickly build Google search queries when you don’t have a lot of information.”
And so the Web Search Query Builder 5 Million Google Sheet was born. … You can try the sheet at https://docs.google.com/spreadsheets/d/1hl0Ku9eqLIcr3piiz5DI_bbeZnQhZP-4eMLRBFLWJ44/edit?usp=sharing . I encourage you to make a copy of it for yourself and play with it. I’ve found it’s useful for getting a quick overview when I run into an unknown concept or name.
An Analysis of Advice About Searching (June 2018)
Daniel Russell with a great analysis of the most important research skills and attitudes, and some advice on how to ask good questions.
(August 2018:) Helpful info from ResearchBuzz: “A glitch in Amazon’s book search”