The Residual Fallacy

For hardcore baseball fans, the “Pythagorean” record, called such for its ostensible appearance to the Pythagorean formula, represents how many games a given team “should” have won over the course of a season. The formula uses nothing but a team’s runs scored and runs allowed and spits out out a winning percentage that very closely correlates with the actual win total of a team. Even more surprisingly, this Pythagorean record correlates closer to the following year’s win-loss record than the previous year’s record does. This strongly suggests, although hardly definitively, that runs scored and runs allowed are a better indicator of team quality than win-loss percentage itself.

Amateur analysts have taken this as a very happy research opportunity. Some looked at how many games a team has won over its Pythagorean record as proof of a team’s strength in ways not measured using runs scored or runs allowed alone. The differential has been used to put numbers to statistical bugaboos like team chemistry, the effectiveness of the coaches, and the usefulness of the running game. However, these methods have fallen out of practice as those variables have failed to exhibit any year-to-year statistical significance in explaining the differential. The only objective factor that has been shown to explain any of the differential is the strength of a team’s bullpen, and that effect is not strong.

We’re lucky that in baseball, unlike real life, we have an amazing degree of data capture and an embarrassment of historical riches, with nearly complete yearly records going back more than 125 years. Retrosheet even has made available full play-by-play data for every season for more than forty years. Every play, every action, of all 162 games, (now) all 30 teams… it is a bewildering accomplishment. This treasure chest of data, an ostentatious auric ensemble of empirics, allows analysts to respond to numerical questions with genuine answers.

Yet, that’s not all. In baseball, every action is discrete. A hitter smashes a home run. A second baseman fields a ball and throws out the runner. Each event is distinct and countable for all to see. Furthermore, the data encapsulates what is important, which is why the pythagorean records work as well as they do. If the subjective, uncountable variables were what really drove a record instead of runs scored and runs allowed, the pythagorean record would not have its predictive power.

The subjective, uncountable variables like team chemistry likely have some effect to a team’s final record, but we have no idea what the effect is. It is now widely accepted to be a fool’s errand to try to tease anymore meaning from the differential. Yet, outside of baseball analysis, the problem of identifying the cause of residuals appears in far more important matters. However, since real-life data rarely possesses the ideal characteristics found in baseball statistics (gratuitous amount of data, countability of actions, importance of objective factors over subjective ones), we cannot take the same logical positivist approach to analysis; that is to say, we cannot ever rely on data to confirm any hypothesis definitively. We may be able to identify correlations for objective causes with some certainty, but we would only be lying to ourselves to think we had captured everything important.

Thomas Sowell has pointed out such presumption in what he calls The Residual Fallacy. The fallacy states that if we control for every objective variable we can find, that persistent statistical significance of a “soft” variable proves that the “soft” variable was directly responsible. But, as stated above, in real life, we don’t have a full picture of everything important. Imagine if in baseball, we only had the number of home runs hit and the average height of the players and were asked to use it to estimate the team’s win-loss record with such information. Without information about the other things that matter, we might pick up on some statistically significant relationship between win-loss record and, for example, average attendance. Would it then be fair to conclude that we have taken everything important into account, and that large crowds cause teams to win more often by cheering?

Yet this is what we do, routinely, including academics and scholars. Courts accept it as evidence. The example that Sowell points to is “proving” racism empirically. It is a fact that, in most industries, if you adjust for age, years of education, marital status, and everything else for which data is easy to collect, Asians make more than Caucasians and Caucasians make more than Hispanics and Blacks. Does this prove racism and put a dollar number on it? Of course not. It is likely that there are additional subjective differences between the groups. This does not mean it’s an INNATE difference or that the groups are somehow better than one another; it means that there probably exist other explanatory variables that analysts cannot capture. At the same time, this does not disprove racism, either. It could well be true that if we could somehow control for literally everything important, race would still be a persistent factor. The point is that we do not know and can’t know in any meaningful sense by throwing everything objective in a regression and seeing what sticks. It just isn’t evidence.

Baseball is a weird case, a rare human event where we can boil down most of what’s important to a few numbers. Still, this doesn’t turn the minds of baseball traditionalists -especially reporters- who spend years with the team and believe team chemistry to be an essential part of a winning organization. They could be well be right; if we could measure chemistry meaningfully, it may explain both some of the differential between the actual and pythagorean records and a reason why the team scored those runs in the first place. Yet, it is strange that some hold standards higher for baseball, where there is evidence that we have captured everything important, than in sociological questions with enormous political ramifications, where believing we no everything is nonsense.

False Probabilities in FiveThirtyEight

NOTE 10/2: I wrote this on 9/20, when it was not nearly as apparent that something had fundamentally changed in the economy. My interpretations on the real effect of the economic downturns, are in fact, absolutely no different than Silver’s at the time.

It didn’t really hit me how popular the blog FiveThirtyEight had gotten until xkcd posted this comic.

Getting mentioned in the same breath as google news and slashdot means something. A political site established without much fanfare this March with an Alexa page rank of 32,175 is more objectively impressive. Site creator Nate Silver’s consistent updates, political objectivity, and clean presentation give credence to his success.

His methodology is what I call into question. The Wikipedia article on the site concisely summarizes,

Polls on FiveThirtyEight.com are weighted using a half-life of thirty days using the formula 0.5^(P/30) where ‘P’ is the number of days transpired since the median date that the poll was in the field. The formula is based on an analysis of 2000, 2004, and 2006 state-by-state polling data.

Hence, his projections are predicated upon a regression analysis using a sampling of historical data. Outside of cliched qualms with polling like selective sampling, most readers looking at the output may be impressed. The site’s tagline, “Electoral Projections Done Right”, seems to be perfectly justified in its self-congratulatory tone.

However, if we care to dig into the number one statistical output provided by the site, each candidate’s win percentage, everything stops making sense. In the passed two weeks, McCain has gone from holding a lead to a scarce 28.6% chance. Something terribly wrong must have happened to cause such an extreme change… but it didn’t. The economy (sort of) blew up, which hypothetically gives some validity to the Democrats, yet it wasn’t anything extreme. It was just a minute, incremental piece of information.

Polling history, even with Silver’s adjustments, are rife with such examples. If it is really that commonplace for the probability of a candidate to get elected to drastically change, then that probability is false. The fact that there is an infinite number of events that could drastically affect the chance of either candidate in either direction between now and November means that any regression model that does not take these impossible-to-predict circumstances under consideration will result in false probabilities.

Gaussian statistics require that extreme variables outside of the model happen extraordinarily rarely. If they occur more frequently, these statistics don’t really work. We may wake up tomorrow to find out that the War in Iraq is suddenly unabashedly succeeding or that McCain is having an affair with a twenty-five year old. Either event may appear to be extremely unlikely, but something extreme, whatever it is, seems more likely than not to occur between now and the election from a historical standpoint.

That damning truth isn’t even really the point. Silver’s model is so flawed that something absolutely meaningless caused a massive change in his projection. If you owned a restaurant and your accountant told you last week that you had a pretty successful month, only to be told by the same guy two weeks later that you must seek bankruptcy protection to continue to operate, there is something wrong going on. That is exactly what Silver tells us in his Electoral Projections Done Right.

I can sit here right now and tell you a better projection than Silver by giving you a number off the top of my head. Obama has a 55% chance of winning and McCain has a 45% chance. If I had it in me, I would fiddle with it slightly each day based on the news and yes, the polls. It wouldn’t change that much any given week, perhaps inching upwards in one direction or another over time. I wouldn’t tell you two weeks ago that McCain would win and now say he has a one in four chance now without anything happening

The polls tell us how much people like either candidate any given moment. They say nothing about what may happen between now and November. Silver’s estimates may accurately capture the current state of events, but, regardless of what his adjusted R^2 says, the current state of events may only be 20% of what matters. Considering the unseen and unpredictable stabilizes the caprice of any projection and pushes it towards the center. Assuming luck and closeted skeletons are of equal likelihood for either candidate, the “real” chance of each candidate getting elected is much closer to 50% than Silver presents it as. The possibility that McCain said something especially denigrating to the Vietnamese or that Obama said something vicious about Middle Americans is real. Either event would nearly guarantee the success of the other candidate. The existence of both possibilities, along with numerous other hypotheticals, pushes the underlying win percentage for both candidates towards 50%.

Even if you think that McCain is more likely to have closeted skeletons because he’s old or Obama is more likely because his middle name is you-know-what, that does not invalidate my fundamental point. An unequal distribution of potential closeted skeletons pushes the probability in one direction, but it stays there. If Obama’s “true” likelihood of winning the election is 80% because McCain is a mean stuffy Republican and subsequently an inevitable hypocrite (or whatever), any change in the following weeks should be no more than percentage points you can count on one hand. It is not, as Silver’s projections assert, a random walk between 45% and 75% depending on how much sleep the voters got this week.

My criticism would not be so harsh if Silver at least clearly presented the essential analysis of variance of his projections. If he provides an explicit caveat that his data suggests that any given variable is only viable to a certain level of confidence, readers willing to delve into such information can backwards-engineer the probabilities Silver asserts. At that point, he would only be committing the same sin that pretty much all social scientists make, that being, to fit the square peg of Gaussian statistics into the round hole of asymmetric, (probably) Mandelbrotiam statistics. Electoral Projections Done truly Right would consider both.

My criticism of FiveThirtyEight isn’t particular. Silver isn’t just self-aggrandizing in his tagline. Most projections aren’t nearly as rigorous as he is. When you hear someone use some seemingly elaborate statistical model to project, it’s important to remember the track record of Gaussian statistics when applied to organic and an uncertain future. Walking up to an informed moderate and asking what her intuition is on the issue or averaging the intuitive opinions of many is far more realistic insisting on empirics when none are feasible. Numerics are not analytical simply because they are quantitative. They first need to be valid.

Rational Opinion-Forming

Uncertainty differentiates opinions and facts. In truth, there always exists a degree of uncertainty in any assertion. The overly skeptical among us point out that the only reason why we believe China exists is that other people have told us it does, and even if we traveled there, you cannot know that the place we actually went to was China, per se. The blurred line of where facts end and opinions begin further confound the question. Are “facts”, like the existence of China or photosynthesis, only those for which there is no disagreement outside of the solipsist circles? Can we also include historical events, such as the Appollo Moon Landing, which a sizable portion of westerners may disbelieve? Where exactly one draws the line is immaterial so long as one applies it consistently. The important thing is that uncertainty differentiates fact and opinion.

Let’s say you’re “line” is 99.5%, which would mean you are okay with believing something you take as fact to be false one in two hundred times [aside- that isn't quite mathematically true, as only the marginal -i.e. those that have exactly a probability of 99.5%- would be wrong one in two hundred times, but that is only important pedantically. this is for illustrative purposes.]. The probabilities to which you’ve assigned a theory, whether it is 99.9999999999% that the world is not flat or 75% that Obama would be a better candidate than McCain, have a specific statistical definition. They are known as a subjective, or Bayesian, probabilities. Even if you do not explicitly assign a number to how sure you are of any opinions, a truth-seeking mind invisibly thinks in such a matter.

This comes into conflict with how many pseudo-intellectuals view logical fallacies. Certain fallacies, such as guilty by association, appeal to the majority, appeal to authority and correlation does not imply causation do not firmly push an argument in one direction or another in any circumstance. However, this in turn does not mean that they should not affect your view at all. Subjective probabilities are all about bringing all relevant information into your mind, assigning the information weights, and coming to a conclusion. The reasonably strong (although not perfect) scientific consensus on global warming means something, even to an informed adult who can understand the literature and formulate a real opinion. The logical fallacy tells us wisely that the weight shouldn’t be infinite, but Bayesian inference tells us that it musn’t at the same time be zero.

There are times when all the evidence we’ve got in either direction are functions of intuition and “fallacies”. Proving or disproving the existence of God provides a watershed of such arguments that, while not argumentatively rigid, do mean something. Rather than listing them, I’ll point here and here. How much weight an individual places on each should determine the direction of their belief, which in turn results in a subjective probability.

The “wobbliness” of each proof does not necessitate agnosticism, but is very instructive of the uncertainty we share in forming any opinion at all. Analogously, this wobbliness appears in questions such as abortion, where we must weigh the fetus’s right to life against the woman’s right to her own body, whatever either of statements really mean. It doesn’t imply that we are to form no opinion whatsoever.

Informally, when we formulate opinions internally, some logical fallacies stop being fallacious. However, there are certain heuristics that appear to hold subjective meaning, but in reality mean literally nothing. Rather than bits and pieces of extraneous information that could conceivably point to truth, such as invoking the opinion of the masses, they are lies. Primarily,

1. Confirmation Bias. To selectively choose history and events that match preconcieved notions, rather than forming opinions through empirics or reason. Thomas Sowell refers to confirmation bias in the scope of empirics as “a-ha statistics”, providing the excellent imagery of a reader perusing the entirety of the newspaper for the few facts that he can point to and say, “a-ha, I knew it all along!”. Evidence to the contrary is subsequently considered chance.
2. Nassim Nicholas Taleb’s “Narrative Fallacy“. Wikipedia gives a very incomplete reference, so I’ll go into more detail on this one. It is the notion that humans look for history to fit into an easily explained narrative, rather than an inconcievably complex entity whose infinite factors and caprice bend in whichever direction for no other reason than they do. Taleb cites, as a starting point, an Italian academic who enjoyed one of Taleb’s earlier works and describes how well it verbalizes the role of randomness rather than reason in success. He then goes on to explain how Taleb could only see this through his upbringing in Lebenon rather than white, Protestant America. Taleb initially agrees, only to realize that such an explanation is ironically against the very role of randomness he argues. Locking onto a single explanatory variable, rather than accepting the existence of multitudes of reasons, gears an individual to a singular, simplistic understanding of what is true, one inflexible to any further incremental information. By perversely believing that whatever chosen variable is the entirety of the truth, he closes himself off to any further information.
3. Monday Morning Quarterback.  Taking past events as inevitable and providing baseless, haughty commentary. By itself, it isn’t all that dangerous besides being absolutely asinine, but when combined with a couple shots of confirmation bias, truth gets thrown out the window altogether. If you go through life thinking that everything is predictable, selecting for the instances that you may have possibly predicted it, all conversation stops and everything in the future becomes an exercise in reinforcing the opinions you already had.
4. Demonization. By ascribing the viewpoints of another to evilness rather than the argument they actually make or resort to pure epithets, no one is getting anywhere. If you actually start believing such perverions of truth (see: Bush), you’re going to think perpetually that half the country is evil and/or stupid. To categorically demonize a point of view is to shove your head in the sand.

These four aren’t the only failures in forming personal opinions, but they are the big ones. Their commonality is their desire to simplify the world into discrete, predictable events inevitably falling in tune with the grand score of life, rather than the cacophony we actually live. Evaluating events and facts as they occur, rather than fitting them to something we can readily understand, is the only way Bayesian reasoning can actually work.

There isn’t anything wrong with taking a piece of information that we honestly believe under consideration. To stuffy logicians, these pieces may be irrelevant, but in the real world of uncertainty, they aren’t. Yet, when our minds stop doing the interpreting and weighing of these facts, and instead ignore or force new information into preconcieved ideals, we lose everything. Subjective probability is the only way the human mind can understand the world, but even such forgiving standards are blemished by certain caveats.

True Anti-Intellectualism

The press, commentators, academics, and pseudo-philosophers slur other points of view as “anti-intellectual” when those views fail to conform to the consensus of science, academia, or the mainstream understanding of either. While even serious analysts may conflate such contrary points of view with the naive irrationality of the “flat-earth” crowd, that characterization both demonizes intellectuals who have taken an unpopular view and ignores the greater evil- proudly true anti-intellectuals.

Set aside your opinion of the views of conservatives, liberals, middle America, Europeans, or city-dwellers. The veracity of any of their political paradigms is not relevant in identifying what I called anti-intellectual. Whether or not those at the forefront of intellectual inquiry are right is ultimately irrelevant when considering the non-intellectual. That is to say, one cannot criticize a non-intellectual Liberal for holding a Liberal perspective. The reasons to be Liberal or Conservative are unclear to society since the people who study the issues for a living cannot themselves come to consensus. Society should not turn non-intellectuals into pariahs as punishment for not holding intellectual comprehension.

While liberal societies should not reflexively mock conservative individuals and vice-versa, that does not imply that the lack of intellectual understanding is somehow a good thing. On all sides of the debate, there exists an attitude that it is somehow bad to provide reasons for one’s opinions. It is almost as if one’s ability to cite sources and facts or the ability to pick out the flaws in another’s argument is offensive. Frequently, conversation makes apparent that there is no reason that a non-intellectual has chosen a point of view, leaving the non-intellectual feeling naked and benighted. Instead of asking themselves why they chose that point of view, the non-intellectual may lash out at the offending criticism as not being supportive or for being overly abrasive.

The desire for communal or friendly support is fine for the trivial, subjective choices of ballet, basketball, or beerpong, but it is quite another for anything that has demonstrative effects, such as the case of politics. In such areas, truth is the only thing that matters, not pleasantries. The idea that it is somehow offensive to demand reasons to justify views and decisions and to provide evidence for one’s own drags society down to an illicit orgy of irresponsibility, irrationality, and emotion.

The inability of intellectuals to come to a consensus is not a moral carte blanche to believe whatever you want to believe. It is easy to hold idly a position knowing that one can go home and in ten minutes find a passage from Bill O’Reilly or Lewis Black, or some other book with a pundit, his expression serious and his arms crossed, on its soft-cover, whenever some waxing intellectual invokes some unknown argument. That ease does not make an idle position justified. This self-affirming dishonesty has two particular flavors: unjustified relativism and simple close-mindedness.

Relativism, while varied in specific meaning and intent, has a general guiding principle. There is no real truth and that anything an individual perceives is equally valid to what anyone else sees. This is a subject that can be rationally debated and discussed intellectually; whether or not the nature of the world is relative is a fact that is objectively true or untrue. Unjustified relativism assumes relativism without citation or argument, insisting that discussing cosmic truth is petty, and that therefore whatever position taken by the individual is sacred. It sidesteps the importance of discussing important issues by citing a vague, philosophical concept brimming in controversy. One’s deep understanding of the intricacies of relativism may justify a general indifference to reality, evidence, and facts proffered exogenously from one’s individual truth, but it does not justify unsupported conjecture ex post.

Simple close-mindedness believes itself to be unmitigatedly and objectively true, but finds the other side of the argument too offensive to take seriously. Those exemplifying this exhibit emotional, negative reactions upon hearing evidence in favor of the views with which they disagree. The desire to reach the goal confounds any discussion of whether or not the goal should be reached in the first place. Learning why intellectuals of the opposition believe what they believe becomes an annoyance, something they don’t want to consider. Such understanding only muddles their ability to construct a counter-factual army of straw men used to guilt a portion of the other side of the argument out of association with the other side. To the close-minded, learning is only learning, and learning is only beneficial, when those pretenses of truth and intellectualism favor the selected goals.

Any argument can retain such specious auspices when truth stops being the rubric of choice. Either flavor of anti-intellectualism mentioned boils down to one factor, pride. Intellectuals know more than us. I personally am not an intellectual. That recognition does not justify my ignorance and blind spots. Instead, that recognition should guide anyone towards filling those blind spots. Failure in prioritizing the discovery of true intellectual discussion renders one anti-intellectual, a personification of boasting ignorance and willful non-understanding. It is unreasonable to demand that we all become intellectuals; it is not to ask that we do not become anti-intellectuals.