An article by Herbert I. Weisberg in the February 2015 Significance (22-27), which is a joint publication of the American Statistical Association and the Royal Statistical Society of Great Britain addresses a concern I have been increasingly having regarding the Gold Standard of research, Randomized Clinical Trials (RCTs). Advances in the sciences and in statistical practice have raised some serious questions regarding their generality.
Actually RCTs are a fairly recent development. For most of medical history medicine was an art practiced by “healers and based on esoteric knowledge acquired mostly through apprenticeship.” The situation began to changed in the 1700s, during the Age of Enlightenment. The scientific method based on empirical evidence about disease and the effectiveness of different interventions began to be applied. However, the article notes that with the exception of a dispute over the wisdom of inoculation to prevent smallpox, the study of medical treatment remained almost entirely qualitative. Even the study by James Lind that eventually led to the use of citrus fruits to cure scurvy would be regarded as a pilot study by today’s standards.
However, at this time, statistical ideas had yet to arrive. Pierre-Simon Laplace was a strong believer in the potential of statistical analysis in various fields including medicine. Laplace’s prescription was primarily theoretical, but it did influence some contemporary medical researchers, One of these was Pierre Louis, who formulated a “numerical method” of assessing treatment efficacy. His approach, applied first in the 1820s utilized simple counts without formal probabilistic analysis.
Two major advances took place in the 1920s that led to the golden age of the RCT. One was the fortuitous discovery of penicillin in1928. This led to a proliferation of new antibiotics that transformed medical practice. The administration of these life-saving treatments was relatively straightforward and depended little, if any, on subtle medical judgment. The effects of these drugs were much less variable with respect to the patient’s response than most traditional therapies. The second advance was in statistics. In 1925 Fisher’s Statistical Methods for Research Workers was published. This provided methodological guidance for RCTs. Gosset’s development of the t-test solved how to analyzed experiments with modest sample sizes. As the article notes, by the 1970s, the methodology of large-scale double-blinded RCT had reached maturity and was broadly accepted as the way to demonstrate the efficacy of a pharmaceutical product, and was mandated by regulatory agencies throughout the world.
It is important that the RCT permits generalization to the population from which the sample used in the RCT was drawn. It does not necessarily generalize to every individual in that sample, or to individuals who belong to other populations. In most studies there are individuals who either didn’t die or recovered from the illness that the drug was intended to eliminate or mitigate. But the RCT requires large samples to estimate statistical confidence.
The advent of epigenetics has refocused the attention on to individuals. Even individuals with the same genetic backgrounds might differ in their response to a treatment because of the way the information was read out of the genome. Genetic differences can determine the efficacy of different treatments. Suddenly the world has become much more complicated. The promise of developing specific treatments for specific individuals has tremendous potential, but is only beginning. There is much to be learned and new techniques for research and treatment will need to be developed. So we must wait and hope.
The important point for readers of the healthy memory blog is that when you read the results of a RCT, the results might not pertain to you individually. This is particularly true in research areas such as mindfulness. You might read that such and such a method was not found to be beneficial. What was found was that the method was not found to be beneficial for the treatment group and for the population from which that group was drawn. But there may have been differences with respect to the research protocol, or to the assiduousness with which certain participants carried out the method. So you shouldn’t necessarily rule out trying the method yourself or some variant of the method.
Tags: American Statistical Association, Epigenetics, Herbert I. Weisberg, individual differences, medical history medicine, Randomizd Clinical Trials, RCT, Royal Statistical Society, Scientific method, Significance, understanding research studies, understanding statistical tests