16 April 2013

For Brain Box "Statistical Power is Truth Power"

In a really excellent essay, "Brain Box" argues that too-much scientific analysis relies upon under-powered data collection:
If you were previously content with 20 participants for fMRI, then perhaps you should recruit 40. If you have always relied on 100 cells, then perhaps you should collect data from 200 cells instead. Yes, these are arbitrary values, but there is nothing arbitrary about improving statistical power. And you can be absolutely sure that the extra time and effort (and cost) will pay dividends in the long run. You will spend less time analysing your data trying to find something interesting to report, and you will be less likely to send some other research down the miserable path of persistent failures to replicate your published false positive.
I would add to this point that too little attention is paid to the actual interpretations attached to the statistical findings. Recall Danziger's comments about statistical analysis in his Constructing the Self:
In the older kind of practice one manipulated experimental conditions in order to test hypotheses about the processes going on in individual psychophysical systems. Now, the direct purpose of experimentation was to make predictions about how certain variations in conditions affected the response of an abstract individual. Because in practice such an individual was statistically and not psychologically real, questions of psychological inference very easily became transformed into questions of statistical inference.... 
It seems to me that statistics are the major problem for philosophers and sociologists of science. Statistics work perfectly but it is not transparent that their object transcends statistics. Putting it differently, statistics would work as well in a Freudian paradigm of self-hood as they could in a neurochemical paradigm of self-hood. I am sure that there are any number of statisticians who could identify even more profound cases.

1 comment: