an important experiment led by a rather prominent young
physicist named Albert Einstein was published. To discover the ratio of magnetic forces to gyroscopic forces
on an electron, Einstein had to infer what the electrons in
an iron bar were up to based on a minuscule rotation their
activity caused the bar to make. His answer was off by a
factor of two, as corrected by more careful, but similarly
inferential, experiments three years later. (What a loser!)
Physicists have a good excuse for huddling under
the streetlight when they are pushing at the limits of
human understanding. But the effect also vexes medical research, where you might think great patient data
is there for the tabulating. The story of the anti-arrhyth-mia drugs only hints at the extent of the problem. In 2005,
John Ioannidis of the University of Ioannina in Greece
examined the 45 most prominent studies published
since 1990 in the top medical journals and found that
about one-third of them were ultimately refuted. If one
were to look at all medical studies, it would be more
like two-thirds, he says. And for some kinds of leading-edge studies, like those linking a disease to a speci;c
gene, wrongness infects 90 percent or more.
We should fully expect scienti;c theories to frequently
butt heads and to wind up being disproved sometimes
as researchers grope their way toward the truth. That
is the scienti;c process: Generate ideas, test them,
discard the ;imsy, repeat. In fact, testing ideas is supposed to be the core competence of most scientists.
But if tests of the exact same idea routinely generate
differing, even opposite, results, then what are we
humble nonscientists supposed to believe?
Ihave spent the past three years examining why expert pronouncements so often turn out to be xaggerated, misleading, or ;at-out wrong. There are several very good reasons why that happens, and one of them is that scientists are not as good
at making trustworthy measurements as we give them
credit for. It’s not that they are mostly incompetents
and cheats. Well, some of them are: In several con-;dential surveys spanning different ;elds, anywhere
from 10 to 50 percent of scientists have confessed to
perpetrating or being aware of some sort of research
misbehavior. And numerous studies have highlighted
remarkably lax supervision of research assistants and
technicians. A bigger obstacle to reliable research,
though, is that scientists often simply cannot get at the
things they need to measure.
Examples of how the streetlight effect sends studies
off track are ubiquitous. In many cases it is painfully
obvious that scientists are stuck with surrogate measures in place of what they really want to quantify. After
decades of dueling studies about whether it was an
asteroid or volcanic eruptions that did in the dinosaurs,
it is apparent that the mineral-deposit evidence is
indirect and open to interpretation, even if the scientists advancing the various claims sound pretty sure of
themselves. Astronomers enlist surrogate measures all
the time, since there is no way to stick thermometers in
stars or to unreel tape measures to other galaxies. Likewise, economists cannot track the individual behaviors
of billions of consumers and investors, so they rely on
economic indicators and extracts of data.
WE SHOULD
EXPECT
THEORIES TO
BUTT HEADS
AS
RESEARCHERS
GROPE THEIR
WAY TOWARD
THE TRUTH.
BUT IF TESTS
OF THE
EXACT SAME
IDEA
ROUTINELY
GENERATE
DIFFERING,
EVEN
OPPOSITE,
RESULTS,
WHAT ARE WE
SUPPOSED
TO BELIEVE?
How reliable are the results? In 1992 a now-classic
study by researchers at Harvard and the National Bureau
of Economic Research examined papers from a range of
economics journals and determined that approximately
none of them had conclusively proved anything one way
or the other. Given that dismal assessment—and given
the great in;uence of economists on ;nancial institutions
and regulation—it’s a wonder the global economic infrastructure is not in far worse shape. (Of course, scienti;c
;ndings that point out the problems with scienti;c ;ndings are fair game for reanalysis too.)
By far the most familiar and vexing consequences of the streetlight effect show up in those ever- shifting medical ;ndings. Take this straight- forward and critical question: Can vitamin D supplements lower the risk of breast, colon, and
other cancers? Yes, by as much as 75 percent, several
well-publicized studies have concluded over the past
decade. No, not at all, several other equally well-publicized studies have concluded. In 2008 alone,
around 380 published research articles addressed
the link between vitamin D and cancer in one way or
another. The ocean of data on the topic is vast, swelling, and teeming with sharp contradictions.
One likely confounding factor is the different ways in
which the studies assessed the intake of vitamin D.
In fact, some of the studies did not measure intake
at all. Researchers simply looked at levels of the
vitamin in subjects’ blood without tracking whether
supplements affected those levels, assuming that
both arti;cially and naturally high levels have the
same effect on cancer risk. In some cases researchers looked at blood levels of the vitamin only after a
cancer had been diagnosed, instead of measuring the
levels before and after. In other cases scientists asked
subjects how many vitamin D pills they took but did
not look at blood levels. The investigators in at least
one widely reported study did not look at vitamin D
blood levels or supplement intake at all. They merely
estimated blood levels based on the sunniness of the
subjects’ geographical locations, since sunlight spurs
the body to produce vitamin D.
The point is not that the scientists running these
studies screwed up. They were probably doing the best
they could with the data they had. We would certainly
be a lot more likely to get a straight answer if someone
would carefully track pill intake, blood levels, and cancer
outcomes in a large population for many years. But such
large, clean studies can take years of planning, fund-raising, and lining up patients, plus a decade or more to
execute. That is why we ;rst get bombarded by years of
weaker studies plagued by the streetlight effect. It sure
would be nice if someone would point that out to us
when one of those studies makes headlines.
