Equivalence Tests - Statistics for Psychology

I'M STILL WORKING THROUGH THIS - IF YOU KNOW WHAT THIS IS ALL ABOUT READ ON AND FEEL FREE TO POINT OUT ERRORS - IF YOU DON'T KNOW WHAT THIS IS ALL ABOUT YOU MAY LIKE TO CHECK BACK LATER

One of the most common misconceptions about p-values is that large p-values show that there is no difference between the conditions that you are comparing. To put it more simply many students interpret a p-value of more than .05 as evidence that the conditions are the same (not different). This is incorrect and reflects an easy fundamental misunderstanding of how p-values are calculated and what influences them. Consider the case where you have two groups that have different mean scores and large standard deviations. In this case the values from each group are highly variable and likely to overlap. Unless you have a large sample size a t-test will be unable to detect this difference; it will return a p-value of greater than .05 even though we know that the true means of the 2 groups are different.

If you want to show that 2 (or more) groups are the same you need to conduct an equivalence test. In an equivalence test your null hypothesis and experimental hypothesis are reversed - your null hypothesis is that the groups are different and your experimental hypothesis is that they are the same. One way to do this is to define two reference points. The first is the point at which you will accept that group 1's score is lower than group 2's. The second is the point at which you will accept that group q's is higher than group 2's. Difference scores falling between these two points allow you to say that the 2 scores are equivalent.

While this sounds straightforward the difficulty comes in setting these two points. While a standard test checks to see if the difference between groups is different to 0 you now have 2 tests checking to see if the difference is greater than and less than some number. Importantly these some numbers do not have a standard definition. You must come up with these points and you must come up with them before running your analysis. Afterall, if you come up with them afterwards what's to stop you picking numbers that will work well for your data set?

While some papers provide example numbers that can provide a useful guideline it is more important to consider why you are running the test and what you are hoping to find. For example if you are comparing 2 drugs for pain relief you might consider a 10% difference in subjective pain relief ratings to be clinically unimportant. This decision would give you your 2 reference points and also highlights an important point about these tests - they are not testing to see whether things are identical, they are testing to see whether or not they are equivalent.

That being said assuming that you wanted to test whether or not two things could be identical one clear item to use to help define your reference points would be your measure of error.