Teasing Information Out of Noise
Bennett Haselton has an interesting
article
at Slashdot about how to develop a “scientific” test for child
pornography.
Given that Slashdot is all about technology and gadgets and stuff, you
might expect the article to describe a new image-recognition algorithm
or something, and wonder how it could possibly work on such a
subjective problem. But he doesn’t; in his proposal, the
“measurement”, if you will, consists of people looking at photos that
may or may not be illegal pornography.
In other words, the problem is that we don’t have a good
scientific instrument that we can point at a photo and have a red
light go on if it’s illegal pornography. All we have is people, who
have to make a judgment call as to whether a given photo is legal or
not.
One problem with this is that we’re asking people to make a subjective
judgment call. And that means they’re likely to get the wrong answer.
So Haselton’s approach is to use people as his instrument, then say
okay, we know that this instrument is unreliable; let’s find out under
which circumstances this instrument fails, and try to counter that.
For instance (he claims), you can take a given photo, mix it in with a
bunch of others that are illegal, and people will condemn the whole
set. But you can then mix the same photo with a bunch of innocent
photos, and people will declare the whole set legal. This tells us
that our instrument is a) unreliable (a photo can’t be legal and
illegal at the same time), and b) affected by context. He then proposes ways to work around this problem.
What I found interesting about this article is that Haselton tries to
apply the scientific method to a highly-subjective area. A lot of
people think science is about labs and test tubes and computer models
and such. But as I’ve argued
elsewhere,
the hardware is secondary, and the core of the scientific method
is really a mindset, and asking the questions “what is the world
like?” and “how do I know this isn’t garbage?”
The other remarkable thing about this article is that it demonstrates
how to tease information out of noise: you have witnesses making
subjective judgment calls on an emotionally-charged subject, biased
prosecutors, biased defenders, and fuzzy legal guidelines. You might
be tempted to throw up your hands and declare that under these
conditions it’s impossible for justice to be served reliably. And yet,
he takes the optimistic approach that no, we can serve
justice, or at least improve our odds of getting the right answer.
It’s a bit like John Gordon’s
summary of coding theory
(aka the biography of Alice and Bob):
Now most people in Alice’s position would give up. Not Alice. She has courage which can only be described as awesome.
Against all odds, over a noisy telephone line, tapped by the tax authorities and the secret police, Alice will happily attempt, with someone she doesn’t trust, whom she cannot hear clearly, and who is probably someone else, to fiddle her tax returns and to organise a coup d’etat, while at the same time minimising the cost of the phone call.
A coding theorist is someone who doesn’t think Alice is crazy.
I often hear that such-and-such problem can’t be approached
scientifically (morality, the existence of God, beauty, whether
something is pornographic or obscene, etc.). A lot of times, though,
it’s because people either haven’t bothered to see how far a
scientific or rational approach can take them, or else they’re unaware
of how much can be done with such an approach, or how
simple it can be. (The reason I don’t make a big deal of whether a
restaurant serves Coke or Pepsi is that my brother and I once tried a taste
test, and I found that I couldn’t really tell them apart.)
It’s also awe-inspiring to think about how far we’ve come, as a
species: not only have we improved our instruments — from naked
eyes to Galileo’s telescope to Hubble — but through statistical
methods, experimental design, and others, we’ve increased the amount
of information we can glean from existing instruments.
Not bad for a bunch of naked apes with oversized brains and a knack
for cooperation.