For 20 years I’ve tried to give helpful guidance to grad students based on the mistakes I made when I was in their shoes: lost my PhD funding, wrote a “Mickey Mouse” dissertation (my advisor’s words), and was “a job market embarrassment” to my department.
I’ve also tried to give helpful guidance to junior profs – again, based on the mistakes I had made when I was in their shoes: 11 consecutively rejected papers, lowest teacher ratings in the school, gay rumors of me on page 1 of the MBA newspaper, denied tenure, lived out of suitcases for 3 years as a visiting professor, rejected again for a tenure track gig, and so on.
For the sake of mental health, when these things happened, I’d try to view them as vivid experiences that would make me a more credible mentor or a more valuable friend at some point in the future. “I’ll know a lot more people in my life who get turned down for tenure, than who get it,” I reasoned. I’ll know more students who strike out on the job market, and I’ll know more colleagues who are hurt by rumors.
So to help more people other than just my immediate students and colleagues, I started this webpage. As seen on the side bar on the right, the goal was to provide “help on how to get your PhD, get hired, and get tenure without making the same mistakes I did.”
For my first column, I wanted to highlight a favorite grad student visitor from 5 years ago. The timing made sense since new visitors come to my Lab around this time of year (they’re usually grad students trying to stretch beyond the range of their advisor’s expertise or interest). When this grad student arrived, I gave her a rich data set (from a field study in a pizza restaurant) and a bunch of interesting unanswered empirical questions about eating in restaurants. After 8 months of dawn to dusk work, she had turned some of these exploratory questions into 4 different exploratory papers. They explore interesting “Who knows?” empirical questions, not preregistered hypotheses.
After publishing this blog about her industrious, a group of researchers contacted us for the data. Because it was not easily de-identifiable, after a couple of emails we mistakenly gave up and ignored them instead of working with them on a timeline to eventually make it available. Shortly after this, they suggested there were 150 inconsistencies between these 4 different papers.
When we read of these inconsistencies, we took this very seriously. We contacted the 4 editors and told them we would rerun all of the statistics, and we asked them if we could publish errata with the new tables. Since then, articles on these inconsistencies have been published in Retraction Watch and in the New York Magazine. This is a very unfortunate situation, and I am so, so sorry for the negative attention it has brought to both us and to the field.
Moving on, here’s what we’re doing to address these issues.
1. For the Journals. All of the analyses are being rerun by a non-coauthor PhD-trained, IRB-approved econometrician in our lab who has access to the confidential data (and who will also eventually de-identify it). As I mentioned, we have already contacted the 4 journals and the new results (confirmatory or not) will be sent to them with errata. We’ll also include data analysis scripts that show exactly how the data was run (inclusion criteria, which measures we used as covariates, and so on).
2. For the Institutional Review Board. The data will be de-identified to the point where our Institutional Review Board believes that their release would not violate the confidentiality agreement with the subjects or the agreement that was made with the restaurant.
3. For all Researchers. The de-identified data, the survey, our analysis scripts, and our errata to the journals will be made available on a public website.
4. For a Concerned Public. After the data for all of the papers has been reanalyzed and we have submitted erratum to the journals, we will be in a good place to address each of the 150 inconsistencies and to explain why they exist. We’re not yet sure where to make this available.
5. For Us and for Other Behavioral Labs. A new set of procedures and standard operating procedures for our Lab will be developed, implemented, tested, and revised. This too has already begun, and its guidance for collecting, analyzing, reporting, and storing data will be very useful in the future in tightening up operations in ways that help prevent this from happening in the future. If other behavioral groups would find this useful, we’d be please to share these on the same website where the data is posted.
Again, I’m so, so sorry this happened. I’ve always been proud to be part of a field that can make people’s lives better. Something like this can tarnish the impact many of us would like our work to have on others. My hope is that eventually all of this will have ultimately helped the field of behavioral research begin to create a) useful new guidelines for collecting field data, and b) standard operating procedures we can all use to confidently move forward with our research without fear of a setback.
Each of those painful lessons I learned as a bumbling grad student and as an earnestly clueless junior professor, are as vivid today as they were back then. That might be why they have been helpful to others since. This lesson will be the same.
Thank you for the kindness of those of you who have reached out to me in this past week. I hope we will cross paths in person.