Replicating Psychological Science in the Classroom

In scientific disciplines, including psychology, we generate knowledge using the scientific method: We develop theories from which we derive hypotheses. We then use a standardized set of manipulations, measures, and statistical procedures to determine whether the hypothesis and its progenitor theory has merit. This is a beautiful approach to acquiring knowledge, one that can take credit for amazing advances in every scientific discipline.

Despite its tremendous power, however, science is not perfect. Manipulations and measures vary in validity and reliability. Moreover, scientists are subject to external pressures to publish a high volume of novel, statistically-significant results. Whereas this “publish or perish” incentive structure is intended to promote innovation and excellence, there’s an unfortunate cost: Scientists do not typically allocate sufficient resources in time and money to replication.

Replication refers to repeating the same standardized set of manipulations, measures, and statistical procedures to see if they yield the same results. A successful replication increases confidence in the generalizability of the original effect. It also enables one to estimate the size of the effect and one’s confidence therein. A paucity of replication efforts is a problem because their absence means that the scientific record contains more false findings than we’d like.

Fortunately, in the last several years, calls decrying the need for more emphasis on replication have gained traction. Indeed, big groups of researchers have spearheaded large-scale replication efforts, including the Reproducibility Project: Psychology (RP:P), the Reproducibility Project: Cancer Biology (PR:CB), and a spate of “Many Labs” replication projects in psychology. However, for many scientists, participating – much less organizing – efforts like these is daunting. Thus, large-scale efforts alone will be insufficient to increase the frequency of replications; we must pursue other options.

According to Frank and Saxe (2012), one option centers on our students. They argue that students are often called upon as part of their educational experiences (e.g., research methods courses) to conceive, design, and conduct studies from start to finish within the space of just a few weeks, often on a topic for which there’s little enthusiasm. But, as Frank and Saxe note, the research that students do in such courses doesn’t have to fit that mold; from their perspective, we should be encouraging our students to conduct publication-worthy replication studies in our classrooms instead. Doing so will simultaneously inspire curiosity and motivation in our students and generate results that have value both inside and outside the classroom.

Based on all of this, yesterday I submitted a small grant proposal to request Tufts internal funds to conduct two publication-worthy replication experiments with the undergraduate students in two of my courses. In PSY 32, Experimental Psychology, we will conduct a replication of a study by Mueller and Oppenheimer (2014). In PSY 42, Emotion Laboratory, we will conduct a replication of a study by Eskine, Kacinik, and Prinz (2011). In the remainder of this post, I briefly describe these two studies and offer a summary of the scientific and pedagogical benefits of this research.

Experiment 1: Mueller and Oppenheimer (2014)
In educational settings like ours, students and professors alike are keen to facilitate student learning. One common strategy adopted by students, and expected by professors, is to take notes during class. In years past, the only option students had was to buy a notebook and a pen and take their notes longhand. More recently, of course, many students opt instead to use a laptop to take notes. Which promotes better learning? Mueller and Oppenheimer (2014) conducted a clever set of experiments to answer this question.

In each experiment, participants watched a prerecorded lecture. Prior to watching the lecture, students received either a laptop or pen and paper so they could take notes. They subsequently took a quiz that assessed their factual and conceptual understanding of the lecture material. Results indicated in two of three experiments that longhand note taking resulted in better conceptual understanding than laptop note taking. In a third experiment, the difference was found only amongst students given an opportunity to study their notes prior to taking the quiz.

There are three reasons why this work is a great candidate for replication in my class. First, it generated significant media attention; pieces were published by the Washington Post, NPR, Scientific American, and other outlets. The presence of media attention indicates that the topic was perceived to be of broad interest to the public. Second, the research has obvious relevance to students and thus has high potential to inspire curiosity and motivation. Third, a replication that draws on a larger sample size will be useful to the field; it will enable us to estimate the size of the effect with greater precision. For these reasons, we will obtain institutional review board approval to replicate Mueller and Oppenheimer’s Study 1 on a total of ~200 participants. With ~96 student experimenters, this target N is feasible within the confines of the spring semester; a small grant will enable us to compensate participants as incentive to participate.

Experiment 2: Eskine, Kacinik, and Prinz (2011)
Consider this scenario: “Congressman Arnold Paxton frequently gives speeches condemning corruption and arguing for campaign finance reform. But he is just trying to cover up the fact that he himself is often bribed by the tobacco lobby, and other special interests, to promote their legislation. How morally wrong is this?”

Eskine, Kacinik, and Prinz (2011) tested the hypothesis that moral judgments like these are influenced by our current emotional state. In their experiment, participants consumed a couple of swigs of a bitter, disgusting drink (Swedish Bitters), a sweet sugary drink (Minute Maid Berry Punch), or a neutral drink (water). Participants then evaluated the moral wrongness of a series of scenarios like the one above. Results suggested that participants who drank the disgusting drink evaluated the behavior portrayed in the scenarios as significantly more morally wrong compared to participants who drank the sweet or neutral drinks.

There are three reasons why this work is a great candidate for replication in my class. First, the main objective of my emotion laboratory class is to teach students how to do research that examines the influence of emotion on behavior; the topic of this study perfectly fits the bill. Second, this study is one of several candidates for replication featured as part of the Collaborative Replication & Education Project (CREP). CREP uses crowdsourcing to encourage replication by undergraduate researchers. According to their web presence, “We aim to leverage these pedagogical benefits to promote replications of important findings. We have composed a list of current studies that we encourage students and instructors to replicate as part of research methods courses, independent studies, bachelor theses, and the like. Everyone who conducts a replication of a study on the list will be encouraged to submit their results to a centralized location where they can be further analyzed by interested researchers/experts. Students and instructors who conduct replications that become part of a published paper will be recognized as contributors on any publications that arise from the data.” It is, thus, an amazing opportunity for students and professor alike. Third, as above, a replication that draws on a larger sample size will enable us to estimate the size of the effect with greater precision. To that end, we will obtain institutional review board approval to replicate this study on a total of ~70 participants. With ~12 student experimenters, this target N will be feasible within the confines of the spring semester; a small grant will enable us to compensate participants as incentive to participate. It will also enable us to purchase cups and beverages for this experiment.

Summary
Replication of these two published studies in my undergraduate research methods courses will yield important scientific and pedagogical contributions. On the scientific side, these replications will help to advance basic science understanding of the impact of note taking on learning and the ways in which emotional states drive moral judgments. I will disseminate the findings in publications and presentations, and make our materials and data freely available for use by other scientists. Pedagogically, our students will obtain valuable, hands-on experience conducting real research whose influence will be felt beyond the confines of the classroom. The chance that this research will be funded is low, but I feel like a winner either way.

Leave a Reply