I was recently directed to an interesting blog post by a pseudonymous blogger who goes by NearCyan titled “Fact-Checking Is Not Easy.” In the post, Mr. NearCyan addresses the “very hard problem” that is fact-checking. I found the post to be mostly agreeable. While I encourage you to read Mr. NearCyan’s blog for yourself, I will briefly summarize some of the key points of his post before offering my own thoughts on fact-checking reform and sound fact-checking principles.
(Note: Returning readers have likely noticed that I use honorifics when referring to living and recently deceased individuals in New Leaf Journal posts, Victor V. Gurbo exempted. Although Mr. NearCyan uses a pseudonym, I saw no reason to diverge from that standard here.)
Mr. NearCyan on Fact-Checking
Mr. NearCyan listed five difficulties inherent in fact-checking. I encourage you to read his blog in full, but I will summarize the key points below.
First, Mr. NearCyan stated that there is “no such thing as an unbiased fact-checker.” I mostly agree, but as I will discuss later in the essay, there are ways to guard against interpretive bias in fact-checking.
Second, he asserted that even if one assumed arguendo that there is a non-biased fact-checker, the non-biased fact-checker must still decide which content to assess. Mr. NearCyan uses the phrase “someone has to cherry-pick the content.” The point holds even if we state, more benignly, that even the least biased and most diligent fact-checker has a finite amount of time and resources.
Third, Mr. NearCyan asserts that many people will not trust fact-checking, even if it is done well, or at the very least will not consume good fact-checking. While I agree, my essay will focus on principles for sound fact-checking separate from whether there is a substantial audience for it.
Fourth, Mr. NearCyan states that “alternative narratives are important for society.” In short, if a commonly accepted “fact” is wrong, fact-checking may turn into a form of misinformation, suppressing the truth or, at a minimum, delaying its acceptance.
Fifth and finally, Mr. NearCyan asserted that “[t]he narrative cannot be controlled,” notwithstanding the grandest ideals of fact-checkers. I do not entirely agree with Mr. NearCyan here – media, corporate, and political entities often control the narrative quite well by creating conditions wherein those who deviate from the narrative pay a social acceptability fee. My views on this subject, however, are mostly outside the scope of the instant essay. I indirectly dealt with some of the reasons for which I disagree in my essay criticizing the use of social pressure and propaganda to intimidate people from referring to the Wuhan virus as having come from China.
Which Fact-Checkers Are We Examining?
As I will discuss in detail further in the essay on fact-checking reform, a good fact-checking system must clearly define in advance how it chooses which statements and assertions to assess. Similarly, a good assessment of fact-checking must clearly define which fact-checkers are being assessed. For the purpose of this essay, I am thinking of mainstream media fact-checkers with wide dissemination – such as national newspapers (e.g., Washington Post, New York Times), other similar news entities (e.g., Associated Press, Reuters), entities devoted entirely to fact-checking (e.g., PolitiFact, Snopes), and even some well-known fact-checkers that have an avowed partisan bent (e.g., Huffington Post, Daily Caller). For much of the essay, I will refer to these fact-checkers collectively as the “fact-checking industry.”
Diagnosing the Failures of the Fact-Checking Industry
Mr. NearCyan ended his essay with a touch of humility, conceding that his content “might not be particularly insightful and is almost too political…” His post was neither not insightful nor too political, but I quote this passage not only to pay a compliment, but to state for purpose of full disclosure that I am less concerned about being “too political” in my post. For that reason, to the extent I may cite to a point that Mr. NearCyan made with reference to one of my own positions on the problems with fact-checking, I make no assumption that Mr. NearCyan would necessarily agree.
Individual and Employer Bias Colors Fact-Checking
The fact-checking industries’ fact-checks are too often colored by the bias of the individual fact-checkers and their employers. For example, to the extent Google and entities under its umbrella perform fact-checks of any sort, consider Google’s history of censoring certain content that is critical of the Chinese Communist Party.
Overt Evaluation Bias
The most obvious way that fact-checking bias manifests itself is in how statements are evaluated. Partisans of one side or the other will often highlight how a statement of questionable veracity by person A is adjudged “true” or “mostly true” after the purported fact-checker appends context that was not in the original statement, while a similarly questionable statement articulating a different perspective by person B is adjudged “false” or “mostly false,” often with the introduction of more carefully selected “context” by the fact-checker. As I will discuss further, sound fact-checking must curtail the use of grades of truth and out-of-statement context to minimize this sort of overt bias.
The latter point – when a speaker is to be given the benefit of the doubt or with the help or harm of additional “context” – is a sort of selection bias. The fact-checker can make full use of a tiered rating system and his or her authority to dispense “context” in order to push certain classes of statements into the passing categories and other classes of statements into the failing categories.
Selection-Bias: Whose Statements are Checked?
Selection-bias is also exercised in two additional ways. First, and perhaps most obviously, fact-checkers must choose which statements to assess and which statements to pass over. Imagine a situation where politician A and politician B are both similarly slippery, and each say 10 things per week that may be amenable to fact-checking. If a fact-checking entity consistently checks five statements from politician A and one statement from politician B each week, readers may come to believe that politician A is less trustworthy merely because he or she seems to say so much that needs to be checked. Conversely, if the fact-checker cherry-picks only false statements from politician B while consistently choosing three out of five true statements from politician A, politician B may appear to be a pathological liar while politician A is just your typical politician.
Selection-Bias: What Kinds of Statements are Checked?
Another sort of selection-bias is exercised in with regard to the types of statements that are fact-checked. The statements that are most amenable to fact-checking are those statements that make a clear assertion about something that has already happened. For example, if a politician says that the current official unemployment rate has changed By a specific percentage between two specific points in time, a fact-checker can easily determine if that is true. Conversely, some statements about the past are mere opinions. If a politician says that a group of people are doing better than they were before, but does not index that to any specific metric, the statement is not generally amenable to fact-checking – and any attempt to assess the statement invites the fact-checker to selectively introduce “context” to make the statement either true or false. Finally, statements about things that have not yet happened are generally not amenable to fact-checking unless the statements contain an assertion about something that has already happened.
Editorializing Masquerading as “Fact-Checking”
Statements that are less amenable to fact-checking are simultaneously most amenable to editorializing under the guise of fact-checking. Thus, if a fact-checker focuses on statements that do not include a clearly and objectively checkable assertion, the fact-checker turns into an editorial columnist. There is no problem with editorializing – we still study The Federalist Papers and the Anti-Federalist Papers today (that is not to suggest that current commentary is on par with those essays, of course). One can more easily affect the narrative surrounding an issue with opinions than with facts. Placing opinions under the guise of “fact-checking” is an attempt by the fact-checker to give his or her own opinion or commentary a greater status than other opinions and commentary by implicitly asserting that any person who disagrees ignores or does not respect the “facts.”
Pertinent Case: “Fact-Checking” Obvious Satire
I am writing this essay in broad strokes, and thus I do not want to become mired in too many specific fact-checking disasters. But there is one case that is too good an example to let pass without mention – the case of Snopes and Babylon Bee.
The Babylon Bee, for those who do not know, is a Christian satire site with humor that often has more appeal to a conservative audience and people who do not like Joel Osteen. It describes itself as “the world’s best satire site, totally inerrant in all its truth claims.” The Babylon Bee’s tagline on Twitter is: “Fake news you can trust.” The vast majority of its headlines, like the secular and left-wing Onion, are obviously satire.
Certain fact-checkers have taken it upon themselves to perform fact-checks of The Babylon Bee’s content, most notably Snopes. Whether Snopes’s pathological fixation with The Babylon Bee comes from a genuine concern that the rubes are unable to understand satire or a conviction that only The Onion and comedians of certain political persuasions should be permitted to engage in satire does not change the effect of this purported “fact-checking.” In Snopes’s decision to fact-check conservative satire while omitting The Onion, The Comedy Central lineup, and similarly aligned comedians, it seeks to send a message that The Babylon Bee is trying to deceive whereas other satirists and comedians are not. Furthermore, even if a small number of people mistake openly satirical stories for truth, not every error is the province of fact-checking.
While The Babylon Bee has turned Snopes’s attention into a running joke of its own, Snopes’s selective fact-checking of one satire site is one of the best, and more humorous (albeit simultaneously depressing) examples of every bad fact-checking trend that I noted above.
Why is the Fact-Checking Industry a Mess and In Need of Reform?
Diagnosing why the fact-checking industry suffers from a parade of maladies is mostly beyond the scope of the instant essay. However, I would be negligent to not acknowledge and briefly discuss the question.
I suspect that the growth of “fact-checking” owes itself largely to diminishing trust in legacy media and to the growth of social media. Regarding the former point, as people began to treat ostensibly objective reporting as opinion-based, or at least something to question, many outlets decided to venture into “fact-checking,” wherein it could confidently separate “fact” from fiction in a way that ordinary journalism could not. This “fact-checking” serves not only to support traditional journalism and reporting, but it also serves as a place to repackage certain stories in a way to give them more authority. The rise of the internet and social media increased the value of stories that could be summed up succinctly, perhaps in a search engine description, tweet, or Facebook post. What kind of content is easier to package in this way than content that can be summed up with the word “Fact” or “Fiction.”
Fact-checking enterprises with entirely benevolent origins may fall victim to the same sorts of problems that plague traditional media. These problems may derive from or reflect the influences of preferences of corporate ownership, the political sensibilities or social and cultural biases of editors and reporters, or the outlet’s being used as a clearinghouse for motivated stories from foreign and domestic actors.
I generally exclude from this analysis fact-checkers at overtly partisan outlets. While these fact-checkers are capable of both sound and flawed fact-checking, readers generally assume some level of bias in what they choose to check and how they check it. Outlets that present themselves as impartial but are not present a far more pernicious problem that will have to be left for another day.
Principles for Sound Fact-Checking and Proposals for Fact-Checking Reform
In the following sub-sections, I will list some basic principles for reforming fact-checking, entirely separate from the merits or demerits of current fact-checkers. I do not suggest that if a fact-checker were to perfectly follow my proposed principles, its assessments would be perfect and free of bias. The purpose of the principles is to outline a clear fact-checking process that readers would be able to understand and to use in evaluating the fact checker’s conclusions. As Mr. NearCyan noted, there is no perfect system to solve every fact-checking challenge, but I do think that a consistent and transparent process can serve as the basis of sound fact-checking.
I. Articulate the Purpose of the Fact-Checking Enterprise
To start, each fact-checking enterprise should clearly articulate its purpose, in other words, what it is seeking to accomplish through fact-checking. If the purpose itself may imply some sort of selection-bias, this statement should make that clear.
For example, Newspaper Y’s fact-checker may commit to only checking the veracity of statements that appear in the newspaper’s own stories in order to provide readers of the paper with more information about the news articles. This is a selection-bias in and of itself, and if what the newspaper chooses to report has a slant, then so too will the fact-checker’s fact checks. But by making clear the limitations under which the fact-checker operates, readers will know going in that the fact-checker is only choosing from a limited set of statements. The same applies too for a partisan fact-checker.
Potentially ambiguous statements must also be clarified. For example, imagine a fact-checker that claims to be combating “misinformation.” What kind of misinformation? There is all kinds of misinformation in politics and on the internet. Broad sounding statements, without specifics, are subject to abuse, especially when applied to ideology.
II. Clearly Explain How Statements are Chosen for Fact-Checking
Principle II largely follows from Principle I. While some selection-bias is necessary, a fact-checker can increase its credibility by clearly explaining how it selects statements and assertions to fact-check. In so doing, the fact-checker should define the universe of individuals, entities, and topics from which it chooses subjects to fact-check. In so doing, the fact-checker discloses its selection-bias and gives the reader much needed context on how to evaluate the fact-checker’s content.
III. Clearly Define Which Categories of Statements are Chosen
It is not enough for a fact-checker to only specify whose statements are being checked; the fact-checker must also specify what kinds of statements it will be checking. For example, let us imagine a careful fact-checker that limits itself to statements most amenable to fact-checking – statements that can, at the moment of fact-checking, be clearly evaluated as true or false based on something entirely objective. The fact-checker could explain, with illustrative examples, the kinds of statements it will evaluate and the kinds of statements that are beyond the scope of fact-checking.
To use an illustrative example of my own, the fact-checker may evaluate a politician’s asserting that the official unemployment rate is lower at point X than point Y, but would refrain from evaluating the politician’s opinion in the same breath that American workers are doing better at point X than point Y. The reason for this is that the first statement can be checked against an objective metric – the official unemployment rate, whereas the use of the word “better” in the second statement is subjective, and not necessarily tied to any single objective metric.
If the fact-checker considers the above example too strict or limiting, it should explain its broader system from the outset. In so doing, the fact-checker should explain how it will evaluate statements and assertions that use amorphous terms and for which determining the truth or falsity thereof may not be so easy as pointing to a specific metric.
The point of Principle III is not necessarily to limit fact-checkers to what I would consider to be their most appropriate place, but rather to ensure that the fact-checker is transparent about its process with his or her audience in order that the audience can understand how the fact-checker chooses statements and then evaluates said statements.
IV. Clarify How Sources for Evaluation are Chosen
In his lucid essay, Mr. NearCyan noted that what may be accepted as a “fact” at one time could be shown to be entirely false at a later time. In one pertinent example, he noted that early information coming out of the World Health Organization about the current Wuhan coronavirus outbreak turned out to be incorrect. Whether one agrees with me about why that was the case, there is no question that the World Health Organization released information about the virus’s transmission that was incorrect, and that “fact-checkers” performed fact checks based on this erroneous information.
Thus, any good fact-checker should loosely define, from the outset, how it chooses sources for fact-checking. I say “loosely” because any fact-checker that checks a wide array of statements is likely to need some flexibility in choosing its sources. But nevertheless, a fact-checker can, in a broad sense, explain its general principles for choosing sources to check specific statements.
To use the World Health Organization case as an example, the fact-checker could list among its sources “United Nations organizations.” Thus, when the fact-checker refers to the World Health Organization as an authoritative source, readers will know that it is among the sources that the fact-checker committed to studying from the onset. If information comes out suggesting that the World Health Organization, or any other source, might not be entirely reliable or impartial, readers can take that into account in evaluating whether its position on a specific matter is trustworthy. That is, the reader can understand the fact-check as concluding that, “based on the position of the World Health Organization, this statement is false,” rather than it being false in some cosmic sense.
Specifying principles for choosing sources at the onset guards against the perception the fact checker cherry-picked sources to tip the scales in certain fact-checks. Furthermore, if the fact-checker has a certain slant that results in its choosing sources with a similar slant, it can square that with the readers from the beginning.
V. Clarify Whether, and How, the Fact-Checker Adds “Context”
The fact-checking industry loves its “context.” Context is often necessary, but in the realm of fact-checking, it is often a portal to editorializing. Public officials often make statements of questionable veracity but do so in a way that allows supporters to say “but he/she really meant Y.”
Rather than simply evaluating statements that refer to some sort of objective measure, fact-checkers will add “context.” To use an example, let us return to the classic unemployment rate case.
Here, a fact-checker is evaluating a politician’s statement that the unemployment rate is lower now than it was at point Y. The fact-checker decides, based on what the politician said, that the politician was referring to official unemployment statistics. The fact-checker can provide some benign context, such as explaining where official unemployment statistics come from before checking to see if the politician’s statement is true on its face.
However, the fact-checker can also add context beyond what the politician actually said. For example, the official unemployment rate does not account for everything, such as individuals who dropped out of the workforce entirely. Thus, a fact-checker may be tempted to add to the politician’s statement, saying that it may be true that the official unemployment rate is lower, but it may not be if we account for people who dropped out of the workforce.
The problem with the latter fact-check is that the politician may have, in fact, only been referring to the official unemployment rate. Perhaps this was intentional – the politician may be well aware that the official unemployment rate supports his or her rhetorical case more strongly than the actual number of unemployed, but the fact of the matter may still be that the politician made no assertion about the latter. In that case, the fact-checker would have exceeded the scope of sound fact-checking by appending “context,” because even if the added context does provide more information about the total number of unemployed, the politician was still referring to the most basic unemployment statistic.
Conversely, if the politician made a statement clearly referring to the total number of unemployed people, the fact-checker may reasonably study more statistics about the unemployment rate to evaluate the statement. In so doing, the fact-checker should remember that sometimes the truth may be uncertain, and that determining the total number of unwillingly unemployed persons may be an inexact science.
VI. Addendum: Separate Superfluous Context from Fact-Check Proper
If the fact-checker is convinced that a given statement absolutely needs context beyond the scope of the statement itself, the fact-checker may provide for, from the onset, a system for separating additional context from the fact-check proper. For example, perhaps a separate individual may add an addendum to the fact-check itself offering some additional information, either purely informational or editorial.. This way, the fact-checker can steadfastly hew to only assessing statements in a narrow way while readers can learn more about the issue separately from the straight fact-check.
VII. Full and Fair Transcribing of Statement Being Evaluated is Necessary
When checking a statement or assertion, the fact-checker should provide the particular statement or assertion in its entirety. While the fact-checker’s additional knowledge about the issue may not be necessary context, the full scope of what the fact-checker is evaluating is absolutely necessary context.
Providing full statements is not only important for transparency and fairness, but also for providing readers with an objective way to evaluate a fact-check. If the fact-checker provides the full statement, the fact-checker can clearly explain how it understands the statement for purposes of the fact-check. The reader can then study the statement him or herself, understand how the fact-checker read the statement, and determine whether he or she agrees with the fact-checker’s starting point. For example, if the fact checker understands a statement about the unemployment rate to refer to the actual unemployment rate, the reader can study the statement for him or herself and determine whether he or she agrees. If the reader thinks that the fact-checker is wrong, and that the politician was making a broader statement about the unemployment rate, the reader can take that into account when reading the result of the fact-check.
Fact-checkers should provide clear rules for how they transcribe statements that are being evaluated, such that readers can be sure that regardless of the perspective of the speaker, all speakers are being evaluated in the same way and being given the same charity in transcription.
VIII: Do Away With the Cute Rating Systems
Fact-checkers need only three ratings: True, False, and Unclear. True and False are obvious enough – is the statement, by whatever objective measure it refers to, true or false? Unclear may be an appropriate evaluation in certain cases. For example, if a politician made a statement about the total number of Wuhan virus cases, but expressly did not tie the statement to the number of positive tests, the only proper evaluation of the statement may well be unknown – for the number of confirmed cases depends on many factors separate and apart from the actual number of cases, such as test availability and who decides to get tested at all. The same applies to a statement about the positivity rate that is expressly divorced from the confirmed positivity rate – the positivity rate may be lower in places where people with no symptoms of illness get tested and higher in places where fewer people with symptoms of illness get tested.
Despite the need for only three ratings, fact-checkers have an annoying tendency to create more ratings. They create grades of truth and falsity, sliding scales, and give certain ratings names that are too-cute-by-three-quarters, such as “Pants on Fire.”
By creating too many ratings, fact-checkers indulge their instinct to editorialize. A statement is either true, false, or unclear. Once a fact-checker opts to add so much “context” to an ambiguous statement that it achieves some grade of truth or falsity on some sort of spectrum, the fact-checker has broken from the restraints of sound fact-checking, and transmogrified the entire enterprise into something else entirely. Worse yet, by giving these rankings “cute” names, the fact-checker stomps his or her feet, demanding that readers look at the fact-checker instead of the fact check. Furthermore, these attempts to be cute combine with the sliding scale of of truth and falsity to tempt the fact-checker to provide commentary rather than basic fact-checks. Not only can the fact-checker put a thumb on the scale to make a false statement only somewhat true or false or a true statement somewhat false or true, the fact-checker can subsequently lend authority to statements that it approve of and derision to statements that it rejects.
Suffice it to say, fact-checkers should use only simple ratings systems and abstain from giving them funny names. Leave the complex ratings and funny names to columnists and other pundits.
Final Thoughts on Fact-Checking Reform
While my views on the contemporary fact-checking industry are less-than-favorable, the above principles provide a way forward for enterprising fact-checkers who are interested in sticking to the facts and the facts alone. I agree entirely with Mr. NearCyan that fact-checking is inherently imperfect and difficult. So too, of course, is the practice of science. In both cases, however, clarifying one’s methodology and starting principles serves not only to standardize the process itself, but to invite readers to understand the process in order that readers can evaluate the results. Fact-checking reform requires a clear, transparent, and forthright fact-checking process. This does not assure that every evaluation will be free from error or bias, but it does ensure that every evaluation will be done through a specific process that readers can follow. Furthermore, by sticking to a particular process, fact-checkers will be better able to identify flaws in their particular fact-checking system (e.g., source(s) that repeatedly prove to be unreliable) and work toward improving the system over time.