P-values indicate confidence on the test, not on the data or finding

When we talk about scientific research, it’s important to make the distinction between “this test result has a 99.99% confidence” and “this will happen 99.99% of the time in the real world”. Looking at just the p-value, we can’t assume that to be a prediction of anything more than the test itself. 

Everything is just a hypothesis, and a test result is just a test result against that hypothesis. p-value of <0.01 says “this test conducted against that hypothesis yielded a result that is very unlikely to be by chance”.  

p-value is just one of the items in the bucket that you have to worry about. These include market factors, behavioural factors, data quality, sampling violations, source bias, research vendor bias, human error and a plethora of other aspects. 

Strong research combines theory, findings and methodology. Methodology includes everything from design and sampling to representing the results. When it all seems to more or less tell the same story, the true confidence is built. When one of it says something else than the rest, the confidence deteriorate. 

Actually the goal is never to have have 99.99% prediction capabilities, but to get to results that are better than absent the method we are using. Like Keynes said it “better to be roughly right than precisely wrong”.

The confidence is never in the test results. The confidence is in our own understanding of the topic we are researching. And the way results from carefully conducted tests (over an extended period of time) reflect on that understanding. 

If you liked this post, please help up by spreading the word or by following us through email. Thank you 🙂