Ratings gathered online vs. in person: Different stimulus sets and different statistical conclusions

Barenboym, Wurm, and Cano (2010) recently showed that significant differences emerged for ratings gathered online and in person. They also showed that researchers could reach different statistical conclusions in a regression analysis, depending on whether the norms were gathered online or in person. In the current study that research was extended. Familiarity ratings gathered online were significantly higher than those gathered in the lab, for a set of 300 potential stimuli. The in-person ratings correlated significantly better with an existing database of familiarity values. It is also shown that under three different grouping methods, online and in-person familiarity ratings produce different sets of stimuli. Finally, it is demonstrated that in each case, different conclusions are reached about variables that have a significant relationship with familiarity. Simulations show that the effects are driven disproportionately by higher intra-item variability in the online ratings. Studies in which stimuli are grouped on the basis of ratings can be affected by the choice of rating methodology.