Experiments & Tampering

 From the always thoughtful Arnold Kling:

W. Edwards Deming distinguished experiments from tampering. With an experiment, you change a process and explicitly compare the results to a baseline. With tampering, you change the process without rigorously examining the results.

For example, in education, most curriculum changes involve tampering. Schools rarely test to see whether a curriculum works.

I once sat next to a high official in the Department of Education, and he was horrified when I suggested experiments in education. “Would you want your child to be part of an experiment?” he asked, incredulously. “The schools do it all the time,” I responded. “They just don’t bother checking to see whether their experiments work.”

It is very hard to make a moral case against experiments that is not also an even stronger case against tampering. But we have a much higher tolerance for tampering than for experiments . . .  Saying that you are conducting experiment implies that you are uncertain. Tampering implies that you know what you are doing. Sadly, people have a higher tolerance for tampering.

You can read the whole thing here.