#11 - JRL 7259
Date: Mon, 21 Jul 2003
From: Gideon Lichfield <gideonlichfield@economist.com>
Subject: Re[2]: JRL 7248 #4: How a journalist gets at the Truth, by Gideon Lichfield

Kirill Pankratov, a physicist, advises us Russia-watchers not to try too hard to explain every detail of the Khodorkovsky affair, because "When the data is too noisy, it is meaningless to explain every datapoint, to adamantly pursue every tiny squiggle on a curve." I'd say the problem is not merely that the data are noisy but that political science is not science. In science you think up a theory to explain things that you've seen (data); then you do an experiment to generate new data, so you can see whether or not the theory holds up. If the new data don't fit the theory, you reject the theory.

In political science you can't do experiments. You can't build a working model of Russia in a laboratory and fiddle with it to see how it behaves. You can't, in short, generate new data. So all you can do is trawl back through the existing data - ie, everything that's happened - looking for the bits that fit your theory.

Inevitably, no matter what theory you've chosen, you'll find lots of data that fit: all those little statements and actions from months ago that had no meaning then but just seem to fall into place so nicely now, if you only look at them in the right light. Worse still, when you find data that don't fit the theory, instead of discarding the theory you discard the data themselves as "not relevant" (easy, since there is no hard and fast criterion of what's relevant and what isn't).

The result of all this is two fatal flaws well known to philosophers of science: the "under-determination of theory by data", meaning that you can think up as many theories as you like because there aren't enough data to sort between them; and the "unfalsifiability" of the theories themselves, meaning that you can't weed out wrong theories using the data because you just ignore any data that don't fit them.

Another result is that it is, in principle, possible to employ an infinite number of journalists and think-tankers in analysing any given situation, since each can come up with his own theory and get paid for doing so. This statement is itself, of course, a theory. I call it the Theory Theory.

If any political scientists out there disagree, let them show us their methodology for analysing all the historical data points in the Khodorkovvky affair and producing the statistical "best fit".

Gideon Lichfield
(The Economist)