For 20 years I’ve tried to give helpful guidance to grad students based on the mistakes I made when I was in their shoes: lost my PhD funding, wrote a “Mickey Mouse” dissertation (my advisor’s words), and was “a job market embarrassment” to my department.
I’ve also tried to give helpful guidance to junior profs – again, based on the mistakes I had made when I was in their shoes: 11 consecutively rejected papers, lowest teacher ratings in the school, gay rumors of me on page 1 of the MBA newspaper, denied tenure, lived out of suitcases for 3 years as a visiting professor, rejected again for a tenure track gig, and so on.
For the sake of mental health, when these things happened, I’d try to view them as vivid experiences that would make me a more credible mentor or a more valuable friend at some point in the future. “I’ll know a lot more people in my life who get turned down for tenure, than who get it,” I reasoned. I’ll know more students who strike out on the job market, and I’ll know more colleagues who are hurt by rumors.
So to help more people other than just my immediate students and colleagues, I started this webpage. As seen on the side bar on the right, the goal was to provide “help on how to get your PhD, get hired, and get tenure without making the same mistakes I did.”
For my first column, I wanted to highlight a favorite grad student visitor from 5 years ago. The timing made sense since new visitors come to my Lab around this time of year (they’re usually grad students trying to stretch beyond the range of their advisor’s expertise or interest). When this grad student arrived, I gave her a rich data set (from a field study in a pizza restaurant) and a bunch of interesting unanswered empirical questions about eating in restaurants. After 8 months of dawn to dusk work, she had turned some of these exploratory questions into 4 different exploratory papers. They explore interesting “Who knows?” empirical questions, not preregistered hypotheses.
After publishing this blog about her industrious, a group of researchers contacted us for the data. Because it was not easily de-identifiable, after a couple of emails we mistakenly gave up and ignored them instead of working with them on a timeline to eventually make it available. Shortly after this, they suggested there were 150 inconsistencies between these 4 different papers.
When we read of these inconsistencies, we took this very seriously. We contacted the 4 editors and told them we would rerun all of the statistics, and we asked them if we could publish errata with the new tables. Since then, articles on these inconsistencies have been published in Retraction Watch and in the New York Magazine. This is a very unfortunate situation, and I am so, so sorry for the negative attention it has brought to both us and to the field.
Moving on, here’s what we’re doing to address these issues.
1. For the Journals. All of the analyses are being rerun by a non-coauthor PhD-trained, IRB-approved econometrician in our lab who has access to the confidential data (and who will also eventually de-identify it). As I mentioned, we have already contacted the 4 journals and the new results (confirmatory or not) will be sent to them with errata. We’ll also include data analysis scripts that show exactly how the data was run (inclusion criteria, which measures we used as covariates, and so on).
2. For the Institutional Review Board. The data will be de-identified to the point where our Institutional Review Board believes that their release would not violate the confidentiality agreement with the subjects or the agreement that was made with the restaurant.
3. For all Researchers. The de-identified data, the survey, our analysis scripts, and our errata to the journals will be made available on a public website.
4. For a Concerned Public. After the data for all of the papers has been reanalyzed and we have submitted erratum to the journals, we will be in a good place to address each of the 150 inconsistencies and to explain why they exist. We’re not yet sure where to make this available.
5. For Us and for Other Behavioral Labs. A new set of procedures and standard operating procedures for our Lab will be developed, implemented, tested, and revised. This too has already begun, and its guidance for collecting, analyzing, reporting, and storing data will be very useful in the future in tightening up operations in ways that help prevent this from happening in the future. If other behavioral groups would find this useful, we’d be please to share these on the same website where the data is posted.
Again, I’m so, so sorry this happened. I’ve always been proud to be part of a field that can make people’s lives better. Something like this can tarnish the impact many of us would like our work to have on others. My hope is that eventually all of this will have ultimately helped the field of behavioral research begin to create a) useful new guidelines for collecting field data, and b) standard operating procedures we can all use to confidently move forward with our research without fear of a setback.
Each of those painful lessons I learned as a bumbling grad student and as an earnestly clueless junior professor, are as vivid today as they were back then. That might be why they have been helpful to others since. This lesson will be the same.
Thank you for the kindness of those of you who have reached out to me in this past week. I hope we will cross paths in person.
My favorite Teaching Assistant applied to PhD programs this year and just got a letter from her favorite school.
Even if you are the world’s greatest TA or the world’s greatest GRE test-taker, getting into a PhD program is a still a gamble. Professors who were great TAs look for great TAs; professors who were great GRE-takers look for great GRE-takers. But some top schools have a 2-5% acceptance rate, and you just don’t know who’s on the admission committee.
Tons of people don’t get in to a PhD program the first time they try. But lots of people who try again, do get in. Here’s what five of them did (and where they went).
• Moved back to home town, joined Toastmasters to improve their
public speaking, and took the GRE three times to raise
their score (University of Miami)
• Worked for $10.50/hour as a research assistant and published
2 papers (Northwestern)
• Taught nights at a community college, worked days as a consulting
firm, and unsuccessfully tried to publish their masters thesis (Stanford)
• Took a job at a marketing research firm and volunteered their
weekends help a local marketing professor with his projects (University
• Learned German at the Goethe Institute, sat in on PhD classes in
Gottingen, and earned a glowing reference letter from a top scholar (Yale)
There’s at least two common denominators here. First, the didn’t give up – they didn’t take their rejection letters as bad omens. Second, they course-corrected. They changed gears in different ways to strengthen their case for the next year. The year I got rejected from seven PhD programs (including Cornell) ended up being a surprisingly fun and empowering year. But every day had an overarching purpose.
The “P” in PhD stands for perseverance. Sometimes that P has to start a year earlier than we want.
Good luck this year. Or good perseverance for next.
Thanks to all you expatriates who filled out the survey.
Here’s what people report happens when they move away from their country:
1. Americans lose weight.
2. Asians gain weight.
3. Europeans gain 6% if they move to the US.
Up Next: Why? Is it smaller portions, more walking, and weird food, or is it cheap food, 57 varieties, and 64-oz Cokes?
Bon Jour! Here’s an international question for any grad student, post-doc, or sabbatical explorer who’s ever lived in a foreign country for more than a couple months. Did you lose or gain weight?
Some people gain weight and some people lose weight. Some move overseas and they walk a lot, and they pocketfuls of funny-looking money on tiny portions, and they lose a lot or weight without realizing it. Or they don’t really like eating yak meat, and they lose a lot.
Others eat with abandon until they need a new passport photo.
It doesn’t matter what country you’ve lived in – US to Brazil, Australia to Taiwan, Finland to the US, North Korea to South Korea – we’d love to hear your story. Ben Missbach (an amazing new visiting scholar from Germany) and I have put together the short survey at the link below. If you can fill it out, we can see if we can put together some rules of thumb to help out the future world traveler in you.
The best writing secret I recently learned was from a Brazilian who didn’t get his PhD until he was about 40.
Yesterday at 5:00, a reporter called me about the topic of food waste. Earlier this year this cool Brazilian coauthor (Gustavo Porpino) and I published a paper about the Food Waste Paradox: Why do low-income people waste more food than middle-income people? We found that the answer is love. Even if everybody is stuffed, the cook doesn’t want their family to feel anxious that the food is gone and bowls are empty. So they often make sure there’s a little extra – even if they have to throw it away.
When he asked me what it was like to work with Gustavo, I said, “Amazing. After we finished all of the analysis, we outlined the paper; he wrote it up; I edited it; we submitted it; and it was accepted with minor revision. Two months start to finish.”
This never happens to me. Why was this guy so amazingly different?
Before going back for his PhD, Gustavo was a journalist. He was used to coming up with cool story ideas that were useful or interesting to a specific audience. He compellingly got this point across in the first paragraph. He built his theory or case with tangible evidence. His results section never steered into detailed analyses that were superfluous to the main story line.
Screenplay writers are taught never to show a “gun on the table” if it doesn’t serve a purpose later in the story. Same should be true with academic articles, but we fill our theory sections and results with loads of these. We almost say, “I don’t know if this is relevant or not, but I’ll include it in case you wanted to know.”
Writing an article like a journalist is a crazy concept. But it just might work.
 Have a powerful lead sentence (vs. “Past research has shown . . .”)
 Show compelling relevance in the 1st Paragraph
 Build your case with tangible evidence (vs. lots-o-cites)
 Stick to the main theme (vs. “could be relevant, but I’m not sure”)
 End memorably
Today the reporter emailed me and asked if there was anything else I could share with him. I sent him this photo and said, “He’s the one with the pumpkin on his head.”
Here's What We're Doing -- See blog "Statistical Heartburn and Long-term Lessons"
Good discussion on this post. Here are two key clarifications to make about data analysis and about the stressed-out workloads of post-docs.
P-hacking and MTurk-iterating isn’t helpful to science, and it’s one of the reasons our lab seldom cites on-line studies. However, P-hacking shouldn’t be confused with deep data dives – with figuring out why our results don’t look as perfect as we want.
With field studies, hypotheses usually don’t “come out” on the first data run. But instead of dropping the study, a person contributes more to science by figuring out when the hypo worked and when it didn’t. This is Plan B. Perhaps your hypo worked during lunches but not dinners, or with small groups but not large groups. You don’t change your hypothesis, but you figure out where it worked and where it didn’t. Cool data contains cool discoveries. If a pilot study didn’t precede the field study, a lab study can follow -- either we do it or someone else does.
About Post-doc workloads. Academia is impatient for publications. It’s the reason why most professors don’t get tenure at their first school (I didn’t get it until my 3rd school). For Post-docs, publishing is make-or-break – it determines whether they stay in academia or they struggle in academia. Metaphorically, if they can’t publish enough to push past the academic gravitational pull as a post-doc, they’ll have to unfairly fight gravity until they find the right fit. Some post-docs are willin to make huge sacrifices for productivity because they think it's probably their last chance. For many others, these sacrifices aren’t worth it.
What follows is a tale of two young researchers.
There’s been some good discussion about this post and some useful points of clarification and correction that will be made with these papers. All of the editors were contacted when we learned of some of the inconsistencies, and a non-coauthor Stats Pro is redoing the analyses. We’ll publish any changes as erratum (and we’ll have an analysis script). This will also give us a chance to change oversights such as cross-citing the papers (they all came out within a year of each other, which led to that slipping between the cracks. Sorry.)
Sharing data can be very useful – like with lab studies and large secondary data sets – and in some instances being willing to do so (or a good reason why not) is a precondition to publishing in some journals. When we collected the data for this study, our agreement to the small business and to the IRB would be that it would be labeled as proprietary and would not be shared because it contained data sensitive to the small town company (sales data and traffic data) and data sensitive to the small town customers (names, identifying characteristics, how many drinks they had, the names of the people they were sitting with, and so on). This is data that cannot be depersonalized since sales, gender, and companions were central to some analyses. (We had explained this when someone requested the data.) At the time we published these papers, none of the journals had the policy of mandatory data sharing, or we would have published these papers elsewhere.
Upon learning of these inconsistencies, we contacted the editors of all four journals to swiftly and squarely deal with these inconsistencies. We told them that we would have the data reanalyzed, and we would write addendums.
Importantly, this field study was not intended to test specific, pre-registered hypotheses. Instead it was intended to learn initial answers to some interesting unanswered questions about eating in a real life restaurant. Does your first bite of a meal influence your attitude more than your last bite? Do you regret eating expensive food less than eating cheap food? Does the person you’re eating with influence how much you eat or like the food?
A PhD econometrician (not a coauthor) is now reanalyzing the data to confirm or refute the published results, and they will be sent to the journal editors. Following this, we will make the addendums, the data analysis scripts, and the data (if we can sufficiently anonymize it to protect our research subjects) available at a link we will add here.
In the end, I think the biggest contribution of bringing this to attention (van der Zee, Anaya, and Brown 2017) will be in improving data collection, analysis and reporting procedures across many behavioral fields. With our Lab, a rapidly revolving set of researchers, lab and field studies, and ongoing analyses led us to be sloppier on the reporting of some studies (such as these) than we should have been. This past Thursday we met to start developing new standard operating procedures (SOPs) that tighten up field study data collection (e.g., registering on trials.gov), analysis (e.g., saving analysis scripts), reporting (e.g., specifying hypo testing vs. exploration), and data sharing (e.g., writing consent forms less absolutely). When we finish these new SOPs (and test them and revise them), I hope to publish them (along with implementation tips) as an editorial in a journal so that they can also help other research groups. Again, in the end, the lessons learned here should raise us all to a higher level of efficiency, transparency, and cooperation.
A PhD student from a Turkish university called to interview to be a visiting scholar for 6 months. Her dissertation was on a topic that was only indirectly related to our Lab's mission, but she really wanted to come and we had the room, so I said "Yes."
When she arrived, I gave her a data set of a self-funded, failed study which had null results (it was a one month study in an all-you-can-eat Italian restaurant buffet where we had charged some people ½ as much as others). I said, "This cost us a lot of time and our own money to collect. There's got to be something here we can salvage because it's a cool (rich & unique) data set." I had three ideas for potential Plan B, C, & D directions (since Plan A had failed). I told her what the analyses should be and what the tables should look like. I then asked her if she wanted to do them.
Every day she came back with puzzling new results, and every day we would scratch our heads, ask "Why," and come up with another way to reanalyze the data with yet another set of plausible hypotheses. Eventually we started discovering solutions that held up regardless of how we pressure-tested them. I outlined the first paper, and she wrote it up, and every day for a month I told her how to rewrite it and she did. This happened with a second paper, and then a third paper (which was one that was based on her own discovery while digging through the data).
At about this same time, I had a second data set that I thought was really cool that I had offered up to one of my paid post-docs (again, the woman from Turkey was an unpaid visitor). In the same way this same post-doc had originally declined to analyze the buffet data because they weren't sure where it would be published, they also declined this second data set. They said it would have been a "side project" for them they didn't have the personal time to do it. Boundaries. I get it.
Six months after arriving, the Turkish woman had one paper accepted, two papers with revision requests, and two others that were submitted (and were eventually accepted -- see below). In comparison, the post-doc left after a year (and also left academia) with 1/4 as much published (per month) as the Turkish woman. I think the person was also resentful of the Turkish woman.
Balance and time management has its place, but sometimes it's best to "Make hay while the sun shines."
About the third time a mentor hears a person say "No" to a research opportunity, a productive mentor will almost instantly give it to a second researcher -- along with the next opportunity. This second researcher might be less experienced, less well trained, from a lessor school, or from a lessor background, but at least they don't waste time by saying "No" or "I'll think about it." They unhesitatingly say "Yes" -- even if they are not exactly sure how they'll do it.
Facebook, Twitter, Game of Thrones, Starbucks, spinning class . . . time management is tough when there's so many other shiny alternatives that are more inviting than writing the background section or doing the analyses for a paper.
Yet most of us will never remember what we read or posted on Twitter or Facebook yesterday. In the meantime, this Turkish woman's resume will always have the five papers below.