In 2017-19, about 18 of my research articles were retracted. These retractions offer some useful lessons to scholars, and they also offer some useful next steps to those who want to publish in the social sciences. Two of these steps include 1) Choose a publishable topic, and 2) have a rough mental roadmap of what the finished paper might look. That is, what’s the positioning, the study, and the possible contribution.
The topics I’ve described here offer one set of roadmaps that could be useful. First, they were of interest to journals in medicine, behavioral economics, marketing, nutrition, psychology, health, and consumer behavior. Second, they each show what a finished paper might look like. They show the positioning, relevant background research, methodological tips, and key implications.
Table 1 and Appendix A lay out an estimate of how much effort it might take to do studies on these topics, and I’ve also estimated what I think the practical impact each research project might have. These are my own subjective estimates, but you might find them a useful starting point if you’re looking for a tie-breaker between two different topics.
I would strongly encourage anyone who’s interested in publishing in these areas to closely follow principles of open science, from preregistration of hypotheses and analytic strategies to open materials and open data. Making specific hypotheses and testing them by following open science principles will be the best next way forward. A good introduction to these principles, along with hands-on advice, is this: Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Hofelich Mohr, A., … Frank, M. C. (2018). A practical guide for transparency in psychological science. Collabra: Psychology, 4(1), 20. DOI: http://doi.org/10.1525/collabra.158
Academia can be a tremendously rewarding career both you and for the people who benefit from you research. Best wishes in moving topics like these forward, and best wishes on a great career.
• Non-native English speakers and shy students have a hard time participating in large classes. Class Previews help them. An hour before each class, I hold a Class Preview that tells people the discussion questions I’ll be asking that day, and then we discuss them ahead of time. Anyone’s welcome to show up.
• For large classes, try to have TA office hours every day (or M-Th)
• Try to be super specific about what you expect for an A assignment, and what your policy is on late assignments and missed classes.
• Rather than having both the final exam and their project due at the end of the semester, I frontload the course with the most work and move their project up to the half-way point.
• Letting students drop their lowest scoring assignment reduces their anxiety and the pressure they feel.
• By having students turn in two copies of their project (one to grade and one to file), it helps ensure the same projects don’t show up year after year.
• I have a screen-down, no-laptop policy. If people say they are taking notes, I ask them to send me the notes after class to see if what I’m teaching is coming across like I hope.
• Class insight cards can give class contribution points to well-prepared, but shy students.
• Extra credit opportunities are great. They help reduce student anxiety.
• If your course is cross-listed with both undergrads and grads, the graduate students will need to do something more (usually about 20-25% more). I also have four specialized class sessions (and a dinner) only for them.
• I try to organize the course into very discrete sections. This way I can frequently review each section in class before I start a new one. This way it very discretely shows how the parts of the course build on each other.
• The Final Project was renamed “Integrative Assignment” and moved to the middle of the semester. They have to work a lot harder early on in the course, but the quality goes up, and their Finals Week stress level goes down.
• You can get the most from a guest speaker’s visit if – before class -- you require students to read something written and published about the guest speaker. (It also makes the guest speaker more of a celebrity). On the day they speak, I will usually show a short Youtube clip of them before introducing them.
• I want the last class session of the semester to be valuable. After they turn in their exam and their course evaluations, I give them the option to leave. For those who stay, I tell them stories of two big lessons I learned in life (the hard way), and how they can deal with challenges in the future. It’s an unusual way to end the class, but it always ends it on a high and memorable note.
Good luck writing a great syllabus, having a great course, and having a triple-great semester.
I went to a super small college in a super small town, and I clearly remember my advisor having me and my college sweetheart over to his home for ice cream on the day I finished my senior project. That reconfirmed to me that I wanted to be a professor. When I was in my master’s program, three of my professors each had us over to their homes for dinner on the last day of the classes. I still remember where I sat and what we talked about.
Over the past 30 years I’ve had way over 200 dinners or parties at my home with graduate students, visiting scholars, summer interns, teaching assistants (TAs), advisees, lab teammates, and so on. The last class session of every graduate class is a dinner, we have welcoming parties for any new person in the Lab (and going away parties when they leave), we celebrate every time we get a major grant by having a party, and when I have a speaker or colleague in from out of town we’ll usually also invite a student or two for dinner with them.
These dinners and parties were always lots of fun. They created of team committment, a lot of good “buzz” (people really wanted to be TAs), people felt very quickly accepted and valued, lab members felt appreciated, and so on.
Looking back, I also think a lot of things could have gone wrong that fortunately didn’t. There was sometimes lots of wine, they’d sometimes last really late, and they were sometimes pretty loud. Sometimes people brought musical instruments and we jammed, sometimes people turned the stereo up to 11, and sometimes neighbors complained. Most of my colleagues tend to not have students over to their homes, partly because of what they imagine could go wrong. They're partly right, but that's largely a shame. There's a solution.
Over the years as I started seeing how different things could spin off the tracks, I gradually started coming up with little rules of thumb I’d use to try and keep things in check. About 10 years ago, I finally honed in on some guidelines that have luckily worked for me. For new summer interns or graduate students, I’ll do them all. For long-term employees or post-docs, I'm more liberal:
 Put a start and ending time on the invitation (like 6:00 to 9:00)
 Have lots of interesting nonalcoholic things to drink
 If there's going to be alcohol, only serve wine
 Only offer the equivalent of 1 glass per person.
(People will often bring other wine or beer, but this usually keeps things in hand)
 Keep everybody on the main floor or on the back deck (and close other doors)
 If there are still females around after 9:00, make sure my wife stays up until they leave
 At 9:00, turn the music off and don’t open any more wine
Now, I used to have none of these rules. I remember when I was a single 30-year old MBA professor, and I was having 26-year old Dartmouth MBA students over to my apartment for end-of-semester dinner parties, there’s a number of things that could have gone wrong. Fortunately they didn't, but these guidelines would have good idea.
My best memories as a student were when professors had me over to their homes. I admired them, and I wanted to be like them. Being invited over, made me feel like part of an inner academic circle. It might not mean anything to some students, but it might also be remembered by others 40 years later.
There once was a legendary, larger-than-life professor. If Marvel’s 137th movie of this year is named UltraProf, it would be based on John Shank. He taught a dry subject (think Accounting), but his charisma and his orchestration of his class made each class session seem like 60-person David Mamet plays. Every class had passion, drama, and some surprising reveal at the end that people still talked about ten years later. Here’s a quote:
As a teacher John was at home at any level, and always brilliant. I could offer the testimonials of others, however, what brought his classroom performance home to me— and it was a performance in the truest sense of the word—was watching him at an Accounting Round Table at Pitt’s business school. He held 50 top financial officers in the palm of his hand while he presented his material. They were busy individuals with many things on their minds and schedules, but not a one left until John had answered their last question. I can assure you that their staying until the end was not out of courtesy (Bernberg 2008).
Although the opposite is true, some believed he was only about style. This is because he wore Brooks Brothers braces, walked with a MVP swagger, drove a Dartmouth green sports car, and he had a runway model wife who was like some VP of Finance somewhere. His office was professionally-decorated with French draperies, super-thick Dartmouth green carpeting, and a massive 18th century French desk which sat in the middle of the room so his desk chair could face the doorway. Even his two huge perfectly groomed dogs were effortlessly well-mannered. On Saturdays he’d come to work, and they’d sit on either side of his desk and face the door. They were like lions on either side of a throne, and he was like Odin . . . or John Wick. His dogs keep eternal vigilance. My dog wets on me and then licks my face.
This was 1992. Because he had about the highest MBA teacher ratings at Dartmouth's Tuck School and I had about the lowest ratings, he let me sit in on his classes so I could suck less . . . so I could learn better teaching strategies and classroom management skills. One Saturday during a Tuck alumni reunion, I stopped by his office and told him I had overheard some alumni who were still talking about what they had learned in a class they had taken with him 10 years earlier.
He looked up over the top of his half-glasses, and said, “That’s what they’re supposed to do. It means I’ve done my job.”
He said his goal isn’t to teach students to get a great first job (or to, analogously, get a high score on the GRE or MCAT), his goal is to teach them to succeed for wherever they will be in 10 or 20 years. Although he got outstanding teaching ratings, he brushed them off by saying that teacher ratings mainly measured the moment – they mainly measured the warm feelings a student had at the time. Ratings might capture style (which he was very good at), but they may not always measure long-term substance.
Last week, the school year ended. A lot of amazing teachers will take their course evaluation ratings and use them to improve their classes for next year. John’s view was that we need to also focus on the long-term impact of our courses.
I regret that I never had the presence of mind to ask him how he did it -- how he knew what long-term impact to aim at. Since he was on boards and did a lot of consulting with upper management, I suspect he taught his courses like he was teaching board members and upper management. That is, when he was teaching, he treated them like they were high level managers. That’s one way to do it.
A second way to try and teach for long-term impact might be to ask. After they graduate, it will be more apparent to them whether your course helped them live a better life (more useful, meaningful, successful, or whatever), and how your course might be improved. It’s easier to get this feedback than you might think. You’ve probably saved your class lists (somewhere) from 5 years ago. You might have their emails, or the alumni office will have both their emails and their snail mail addresses. You can simply ask them.
Ten or fifteen years after I left Dartmouth I was in Boston, and I rented a car to drive up to visit John. I wanted to thank him for being so generous, and I wanted to prove to myself that his office, desk, and dogs were as amazing as I remembered them. There was a different name on his door. I was too late.
I love the idea of trying to teach for a long-term impact. It’s like trying to create long-term memories. I sometimes think I can remember everything John said to me because he was always so intentional with every conversation. Just like he was with his classes.
At the next reunion, if his former student’s aren’t talking about what they learned 30 years ago, they’ll be talking about how hard he tried. That itself was a great lesson.
A close friend of mine believes successful PhD students have three things in common: They're smart, they work hard, and they have good judgment. The secret sauce here is "good judgment." Although smarts and hard work are important, most PhD students would have never been admitted if they weren't already smart, hard workers.
But having good judgment is more elusive. It includes things like knowing what's "interesting" and what isn't, knowing what's worth worrying about (and what isn't), knowing what's important to prioritize, knowing how to solve a people problem, knowing whether to persist on a project or to move on, and so on.
But advising a PhD student to "Have great judgment" is like advising a football team to "Score the most points." It doesn't tell them how. You can't say "Have great judgment" and then say "Next question," "QED," or drop the mic and walk out.
Maybe there's two types of judgment -- technical judgments and nontechnical judgments.
For graduate students, building technical judgment is about learning the whys of research. One way to build better technical judgment is to boldly ask lots of "Why?" questions of your mentor, advisor, or of an older student: "Why did you send it that journal? Why didn't you use a different method? Why did you ask the research question that way?" Most of us shied away from asking technical judgment questions because we didn't want to be irritating or look like we didn't belong. Most professors I know actually like to answer these questions, and they love to see an engaged student step out of a silent huddle.
Developing good nontechnical judgment is trickier. Yet this is the critical judgment you need to troubleshoot how you can be a better teacher, or whether to choose the risky dissertation you want to do versus the safe dissertation your advisor wants. It involves figuring out how to deal with your off-the-chart stress level or whether you should take a job at a teaching college or go into industry. Our tendency as a graduate student is to get feedback from peers in our same year. A more effective one may be to get it from recent graduates or from professors who have seen cases like these and know how they worked out. You can even get nontechnical advice from professors you know in other departments. The best nontechnical dissertation advice I got was from a Medical School professor from my church. It was straightforward, unbiased, kind, and based on lots of students he had known.
As professors, we can help to build better technical judgment by encouraging questions about our research judgment calls, or we can give it as color commentary or as context when we discuss a research project. But again, helping students with nontechnical judgments is trickier. One way to do this is in the third person. This can be by discussing a problem that "their friend" is having or by discussing a relatable case study.
Here's one approach to building nontechnical judgment. I used to teach a PhD course where we'd meet in my home for a casual last class session. The first half of the session would be a discussion about graduate student success and the last half would be dinner. Each student had been asked to anonymously write down a dilemma that "one of their friends" was facing that was being a roadblock to their success. We'd mix these 9-10 dilemmas up, and we'd relax in the living room with a glass of wine and discuss them one at a time. For each one, we'd talk about similar experiences, options, solutions, and so forth. By dinner time, we had a more balanced perspective and some suggested next steps for many of the dilemmas.
Over the years, it seemed that about 70% of these dilemmas were about the same 7-8 issues. These were like the issues mentioned above -- "risky vs. safe dissertation," "stress level," "leave academia," and so on.
Here's a second approach to building nontechnical judgment. Given how similar these dilemmas were from year to year, I wrote up 1-page PhD student case studies that involved slightly fictionalized people who were facing these common problems. These case studies were in the syllabus for the course, and we'd take the first or last part of each class to talk about that week's case study. The common dilemmas faced in your field may be different, but the enthusiasm your students would have in discussing them would probably be the same.
Some people might be born with great judgment, but for the rest of us, it's a lot of trial and error and a lot of asking bold questions. If you're a graduate student, you've got a lot more license than you might think to learn from trial, error, and bold questions. If you're a professor, there's a lot we can do to help them.
How to graduate, get a job, get tenure, solve problems, and make fewer mistakes.