Paper Highlights

Column

Effect Size

Correct Conclusions (Power) vs. Incorrect Conclusions (Error)

Simulation Procedure

Conclusion Flowchart

Simulation Results

Column

Power Rates

Error Rates

Contact the Authors

Udi Alter
udialter@yorku.ca
udialter.com
@UdiAlter

Alyssa Counsell
a.counsell@ryerson.ca
alyssacounsell.com
@AlyssaCounsell

References

References

Altman, D., & Bland, J. (1995). Absence of Evidence Is Not Evidence of Absence. British Medical Journal. 311. 485. https://doi.org/10.1136/bmj.311.7003.485. 

Anderson, S., & Hauck, W. W. (1983). A new procedure for testing equivalence in comparative bioavailability and other clinical trials. Statistics and Communications-Theory and Methods, 12 ,2663-2692.  https://doi.org/10.1080/03610928308828634

Schuirmann, D. J. (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15 ,657-680. https://doi.org/10.1007/BF01068419 

---
title: "Equivalence Testing for Multiple Regression"
author: "Alter & Counsell (2021)"
output: 
  flexdashboard::flex_dashboard:
    theme: flatly
    vertical_layout: fill
    social: menu
    source_code: embed
    navbar:
      - { icon: "fa-newspaper", href: "https://psyarxiv.com/ugc9e/", align: right }  
---

```{r setup, include=FALSE}
library(flexdashboard)
library(magrittr)
library(simglm)
library(tidyverse)
library(haven)
library(hrbrthemes)
library(plotly)
library(greekLetters)
library(MetBrewer)
library(datapasta)
library(DT)

```

```{r initial data, include=FALSE}

simresults <- readxl::read_xlsx("simulation results.xlsx", col_names = TRUE)


simresults$approach <- factor(simresults$approach, levels = c("difference based", "equivalence based"), ordered = FALSE)
simresults$`test type` <- factor(simresults$`test type`, levels = c("DB", "TOST", "AH"), ordered = TRUE)
simresults$predictor <- factor(simresults$predictor, levels = c(1, 2, 3, 4, 5), ordered = TRUE)
simresults$`effect size` <- factor(simresults$`effect size`, levels = c(0, 0.05, 0.1, 0.15, 0.2), ordered = TRUE)
simresults$`sample size` <- factor(simresults$`sample size`, levels = c(50, 75, 100, 250, 500, 1000), ordered = TRUE)

str(simresults)

correctLOA <- subset(simresults, simresults$`effect size` != "0.15" & simresults$`effect size` != "0.2") 
incorrectLOA <- subset(simresults, simresults$`effect size` == 0.15 | simresults$`effect size` == 0.2) 

correctLOA <- correctLOA %>% 
  rename(
    es = `effect size`,
    ss = `sample size`
    )

incorrectLOA <- incorrectLOA %>% 
  rename(
    es = `effect size`,
    ss = `sample size`
    )

```

Paper Highlights {data-icon="fa-highlighter"}
=============================

Inputs {.sidebar data-width=300}
-------------------------------------
Abstract  

Psychological research is rife with inappropriately concluding “no effect” between predictors and outcome in regression models following statistically nonsignificant results. This approach is methodologically flawed, however, because failing to reject the null hypothesis using traditional, difference-based tests does not mean the null is true. Using this approach leads to high rates of incorrect conclusions which floods psychological literature. This study introduces a novel, methodologically sound alternative; we demonstrate how to apply equivalence testing to evaluate whether predictors have negligible effects on the outcome in multiple regression. we constructed a simulation study to evaluate the performance of two equivalence-based methods and compared it to the traditional test. we further developed two R functions which accompany this study to supply researchers with open-access and easy-to-use tools. The use of the proposed equivalence-based methods and R functions is illustrated through examples from the literature, and recommendations for results reporting and interpretations are discussed. For the full paper, R code, functions, slides, and supplemental materials please see https://osf.io/w96xe/ and/or https://github.com/udialter/equivalence-testing-multiple-regression. You can also press the top right "paper" button to be directed to the PsyArxiv preprint.


Column {.tabset data-width=500}
-------------------------------------
   
### Effect Size

```{r}
knitr::include_graphics('effectsize.png')

```   

### Correct Conclusions (Power) vs. Incorrect Conclusions (Error)

```{r}
knitr::include_graphics('table.png')
```   
 
### Simulation Procedure
    
```{r}
knitr::include_graphics('simproc.png')

```

### Conclusion Flowchart
    
```{r}
knitr::include_graphics('flowchart.png')

```

### Simulation Results

```{r}
datatable(simresults)
```

Column {data-width=500}
-----------------------------------------------------------------------

### Power Rates

```{r}

es.labs <- c('\u03B2 = 0', "\u03B2 = 0.05", "\u03B2 = 0.1")
names(es.labs) <- c("0", "0.05", "0.1")

p <- correctLOA %>%
ggplot(aes(x =ss, y = `LOA conclusion rate`, group = `test type`, color= `test type`, text = colnames(correctLOA))) + geom_line(aes(text= paste("Test: ",`test type`, "\n\u03B2 = ",es, "\nn: ", ss, "\nCorrect Conclusion Rate: ",100*`LOA conclusion rate`,"%", sep = "")  ), alpha= .5,size= 1.5, position=position_dodge(width=.1))+facet_wrap(~es, labeller = labeller(es= es.labs))+theme_minimal()+labs(y="Rate of Correct Negligible Effect Conclusions", x = "Sample Size")+ scale_color_manual(values=met.brewer("Isfahan1", 3))+theme(axis.text.x = element_text(angle=45,vjust = 1, hjust=1))
                                                                    
ggplotly(p, tooltip = "text")
```

### Error Rates

```{r}

es.labs0 <- c('\u03B2 = 0.15', "\u03B2 = 0.2")
names(es.labs0) <- c("0.15", "0.2")

p0 <- incorrectLOA %>%
ggplot(aes(x =ss, y = `LOA conclusion rate`, group = `test type`, color= `test type`)) + geom_line(aes(text= paste("Test: ",`test type`, "\n\u03B2 = ",es, "\nn: ", ss, "\nIncorrect Conclusion Rate: ",100*`LOA conclusion rate`,"%", sep = "")  ),alpha= .5,size= 1.5, position=position_dodge(width=.1))+facet_wrap(~es, labeller = labeller(es= es.labs0))+theme_minimal()+labs(y="Rate of Incorrect Negligible Effect Conclusions", x = "Sample Size")+ scale_color_manual(values=met.brewer("Isfahan1", 3))+theme(axis.text.x = element_text(angle=45,vjust = 1, hjust=1))
                                                                    
ggplotly(p0, tooltip = "text")
```


Contact the Authors {data-icon="fa-user-ninja"}
=============================
Udi Alter  
[udialter@yorku.ca](mailto:udialter@yorku.ca)  
[udialter.com](https://udialter.com/)  
[@UdiAlter](https://twitter.com/UdiAlter)



Alyssa Counsell  
[a.counsell@ryerson.ca](mailto:a.counsell@ryerson.ca)  
[alyssacounsell.com](https://www.alyssacounsell.com/)  
[@AlyssaCounsell](https://twitter.com/AlyssaCounsell) 


References {data-icon="fa-bookmark"}
=============================
References  
 


Altman, D., & Bland, J. (1995). Absence of Evidence Is Not Evidence of Absence. British Medical Journal. 311. 485. https://doi.org/10.1136/bmj.311.7003.485. 

 

Anderson, S., & Hauck, W. W. (1983). A new procedure for testing equivalence in comparative bioavailability and other clinical trials. Statistics and Communications-Theory and Methods, 12 ,2663-2692.  https://doi.org/10.1080/03610928308828634

 

Schuirmann, D. J. (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15 ,657-680. https://doi.org/10.1007/BF01068419