Science, as a process and as a body of research, doesn’t rest. One of its strengths is its ability to constantly change and reify the information it produces. Unlike many aspects of the world, in science, being proven incorrect is seen as a virtue.
The struggle to find the best answer means going through more than a few wrong answers. What researchers and scientists fear, however, is incomplete answers. Verifying information is a tricky task, one that leads to some conflict when discussing repeatability vs reproducibility.
This comes on the heels of a troubling study that indicated 68% of surveyed studies had high degrees of reproducibility. An inability to confirm that the findings from one study are applicable leaves big question marks that cause trouble.
Both a right and a wrong answer have value, a shoulder shrug creates problems. Read on to learn more about how these concepts are used and their importance.
Repeatability vs Reproducibility
Let’s start with some background on why this topic matters in general. The introduction touched on it, but it’s fundamental to the matter.
By some definitions, science is a process by which we observe, test, and explain the world. An underpinning of this system is the idea of uniformitarianism. The assumption has to be made, axiomatically, that things operate the same today as they did yesterday and also will work tomorrow.
With this in mind, any study done by one group of scientists will work the same when done by another group. When results cannot be replicated, it calls into question uniformitarianism. Since that is axiomatic to everything else, it’s more likely an error occurred.
Good science reduces error and strives for a complete picture. Sloppy science relies overly much on correlation.
It is from this mindset that researchers publish their findings and data for others to scrutinize through repeatability and reproducibility.
The Crisis
The paper referenced above came out in 2015. The issue of low reproducibility in certain scientific publications isn’t that new. The current scrutiny comes from the increasing influence of psychology and sociology.
These so-called ‘soft-sciences’ are harder to test and record general data. Often studies in these disciplines rely on self-reporting. Self-reporting is a notoriously fussy system prone to wishful thinking from test subjects.
These sciences increasingly affect the world as they are used to justify any number of socio-political stances. Policy creation has enough problems from politicians without the data being soft.
The crisis of reproducibility isn’t about poor quality research, not necessarily. Papers constructed with poor methodology, small sample sizes, and other known weaknesses are more related to the existence of pay to publish journals.
Repeatability
To repeat a study is to, as closely as possible, replicate everything done in the first study. This includes performing the same tests in the same place with the same equipment.
A quick rule of thumb is to think of repeatability as being the same researchers double-checking their findings.
A repeated study needs to get a result as close as possible to exactly the same as the previous attempt. A correlation coefficient is created in this way to indicate the strength of the similarity. The stronger the coefficient, the more the study can be said to hold up.
Repeatability suffers from a few issues when it comes to the veracity of the results. The first issue is that with so many elements being the same from one test to the next, that a result can be the product of a flaw in the equipment, location, or etc and therefore continue to occur.
This can be useful for finding a specific error. For example, if calibrations uncover that something was set wrong. Another would be if the external conditions can be shown to impact the study.
Reproducibility
The difference brought in by reproducibility is eliminating the what-ifs of those flaws. A different team with different equipment attempts to recreate the same experimental tests and arrive at the same result.
Reproducibility provides a stronger correlation that a hypothesis works. The results are not novel or a fluke to a particular lab, the trust level in the uniformitarianism goes up.
In science, nothing can ever be fully ‘proven’, proof is a concept for mathematics only. But a strong correlation with a high degree of repeatability and reproducibility points in a good direction.
Reproducibility also weeds out human and measurement errors. It is far more likely that an effect exists when hundreds of researchers get roughly the same answer than one person that is prone to the same flaw in their thinking and execution.
Correcting for Known Variables
Harder sciences, such as chemistry and physics, are also prone to issues with repeatability and reproducibility. The University of Utah and it’s one successful, but unrepeatable, cold-fusion test comes to mind.
One reason that hard sciences have a higher degree of success in reproducibility is that the variables have less wiggle room. Dealing with the specific impulses of a brain as you get in psychology, sociology, and even neuroscience creates issues.
To combat these problems, the statistical mean is used more than any single sample. However, getting enough subjects and pulling enough data from them has its own set of challenges including observer effects. (A person knowing they are being observed behaves differently than when they are unaware.)
To deal with this issue in a head-on way, some scientists are looking into ways to collect data samples in a repeatable fashion that removes as much observer effect as possible.
The device listed in this article tackles the problem with testing lab mice. The usual process of selecting and testing mice creates extra variables. By removing the variables and flattening the test parameters, the results have increased reproducibility.
Stay Ahead
Information about health and fitness changes constantly. The human-machine is a complex system and every individual brings their own set of unique challenges. These small differences make it even more important that scientists navigate the problems of repeatability vs reproducibility.
Keep coming back here for more news and updates on things that affect your world.