Systematic data collection (Reisberg, ch. 12)

In our daily lives, we often rely on judgment and reasoning heuristics-short-cut reasoning strategies that usually lead to the correct conclusion, but which sometimes produce error. This means we sometimes draw inappropriate conclusions, but these errors are simply the price we pay for the heuristics' efficiency. To avoid the errors, we'd need to employ reasoning strategies that would require much more time and effort than the heuristics do.

For scientists, though, efficiency is less of a priority; it's okay if we need months or even years to test a hypothesis. And, of course, accuracy is crucial for scientists: We want to make certain our claims are correct, and our conclusions fully warranted. What steps do scientists take, therefore, to avoid the pitfalls of ordinary day-to-day judgment and thus to minimize error?

The answer to this question begins with the fact that, in ordinary reasoning, people are heavily influenced by whatever data are easily available to them, and often they're insensitive to the fact that the easily available evidence may not be representative of the broader patterns in the world. Likewise, when people gather evidence, they are often influenced by confirmation bias, and so seek out evidence that might support their views, but do little to collect evidence that might challenge those views.

To avoid these problems, scientists insist on systematic data collection. This rules out arguments based on anecdotal evidence-evidence that has been informally collected and reported-because an anecdote merely provides one person's description of the data, leaving us with no way to determine whether the description is accurate. The demand for systematic data collection also requires us to collect enough evidence, so that we're not persuaded by a couple of salient cases (what the chapter calls "man who" stories).

We also need to collect the data in a fashion that guarantees equal emphasis on the facts that support our hypothesis and the facts that do not. This broad consideration must guide our choice of participants, and our design of the procedure, and how the data are recorded. For example, we cannot rely on our memory for the data, because it's possible that we might remember just those cases that fit with our interpretation. Likewise, we cannot treat the facts we like differently from the facts we don't like, so that, perhaps, we're more alert to flaws in the observations that conflict with our hypotheses, or less likely to report these observations to others.

Scientists also take steps to combat another form of confirmation bias-the file-drawer problem. This term refers to the fact that an investigator might choose not to publish disappointing results (that is, results that don't confirm his hypothesis!); instead, the data are dumped into a file drawer and (potentially) forgotten. To guard against this tendency, scientists periodically seek to gather together all of the evidence on an issue (including unpublished results), and report the full data pattern in a review article that considers all of the findings, not just those on one side of an issue.

Clearly, then, many elements are involved in systematic data collection. But all of these elements are crucial if we are to make certain our hypotheses have been fully and fairly tested. And in this regard, scientific conclusions are invariably on a firmer footing than the judgments we offer as part of our daily experience