Peer Review Review

From a variety of sources, I have learned about this not-at-all recent study. From the abstract:

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables. 

The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices. 

With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.

 While I don't want to get carried away, or anything, this would seem to be a somewhat serious indictment of the peer-review practices of the psychology journals punked investigated by the researchers. Looks bad. Best case scenario is, publishing in these journals is a crapshoot where the odds are an abysmal nine-to-one against; worst case scenario is we're all getting butchered.

However, I was a little disappointed that the study didn't include something in the way of a control group--it seems to me that it would have been a better design if they'd substituted (actual) prestigious institutional affiliations for some of the articles, instead of using all fictitious institutions. And, along the same lines, it would have been a better design if they'd have given the same treatment to a group of papers published in C-level journals--submitted them to high-level journals, half with affiliations with fictitious institutions and half with affiliations with real, prestigious institutions. It seems to me that a study with that design would be a lot more conclusive. (Not to say, 'conclusive.')

(Also, a commenter at Philosophers' Cocoon says that the journals investigated all practiced non-blind review procedures. I'm working from home today, and am unwilling to jump through the hoops I'd need to in order to read the article, so I'm just going to take her word for it. But if that's right, it takes almost all of the "wow" factor away. It's still kind of bad that they didn't recognize the articles as having been already published by them, but if your job is mostly to receive submissions, send them out to review, and deal with the results, it's easy to imagine that you wouldn't catch on to something like that. I, for one, wouldn't be on the lookout for it.)

--Mr. Zero

R for the R & R (Requiem for the Revise and Resubmit)

It occurs to me that it has been something like five years since I've had a "revise and resubmit" verdict from a journal. In that time, I've had several papers accepted "as is," several more conditionally accepted as long as I am willing to make the changes and successfully execute, and a bunch of rejections. Many of these rejections, of course, come with no comments. But when they do come with comments, it sometimes seems to me that the actual comments are basically consistent with an R&R. They say, "a few minor suggestions," or some such thing, not, "there were several serious problems." Often these reports don't say anything specific about what they recommend the editor do—that material usually goes straight to the editor—but one report I recently got was explicit that the referee thought the paper should be R&R-ed. The editor, of course, rejected the paper.

And it seems to me that many of my friends have had the same experience. At least, the ones I've discussed this with. Although this is so unscientific that it's of basically no value whatsoever. Probably shouldn't have mentioned it. 

But I have gotten the sense that, over the past ten years or so, space in the journals has gotten increasingly scarce—witness Nous and PPR's annual six-month submission hiatus—and that this makes editors increasingly reluctant to deploy the R&R, and to reach straight for the "reject" button instead. 

And that's too bad. I like R&Rs, and not just because it's not a full-on rejection. I like hearing that the editor believes in the paper enough to give it another shot. I like getting real feedback from an editor who isn't sending it along just in case I'll find it helpful, but who actually believes that the suggestions will make the paper better. And I like being in a position to discuss and possibly negotiate the proposed changes. I think there's legitimate value in the R&R, and it makes me sad to see it go. If, indeed, it is going. Which, maybe it's not. 

Does that seem right? What am I missing? 

--Mr. Zero

What Made The Difference?

In comments, anon 9:58 asks:

How about a thread where people can post who were successful this season? I don't just mean grad students from Leiterrific departments with 0 pubs and unpublished brilliant writing samples and famous dissertation advisors. I think a thread, specifically with people from departments with little prestige, preferably who were multiple times on the market. What went right this time? Did you do something different, or was did your pubs etc reach some critical mass? How long were you on the market? What did you do in the way of networking? What kind of temp positions (e.g., postdoc, VAP) are more likely to increase your desirability as a candidate?

I think that's an excellent idea. If you were on the market for a while a while before you nailed down a tenure-line job, what do you think made the difference? What's the deal?

--Mr. Zero

My Statistics, 2013 Edition

I apologize for neglecting you recently. I was busy with the usual stuff: teaching, writing, grading, child rearing, trying to keep myself in reasonable physical condition, getting rejected from journals, etc. Time got away from me, and before I knew it, it had been forever since I'd posted anything. Sorry about that.

Here are my stats for the '12/'13 job market season:

• Applications: 43
• APA interviews: 0
• Skype interviews: 0
• Phone interviews: 0
• On-campus interviews: 0
• Nibbles of any kind: 0
• Servings of beer/cocktails: ≈∞

Obviously, this was incredibly shitty. There was the growing realization that I wasn't getting any APA interviews, which at least had the upside that my presence would not be required in Atlanta. Then there was the growing realization that I wasn't getting any post-APA delayed-schedule first-round interviews, either. Or any straight-to-campus-visit type interviews. And that none of my late-season applications were going to pan out, either. Pure shit.

And it's not as though I was an unprepared or inexperienced job-seeker. I based my application materials on the materials I used the previous year, when I had what I thought was a pretty good number of interviews. I went over everything carefully myself before I sent it all to my grad program's placement director--we had a new PD this year, and he wanted to go over everything with a fine-tooth comb. He was amazingly, terrifically helpful--we talked on the phone for an hour while he went over everything in my dossier line by line. Then, in post-mortem, he and my dissertation director went back over everything again. Everything looked good. No suggestions.

I've talked about this with a lot of people, and thought about it a lot. I'm inclined to suspect that it was just bad luck, but I also worry that I'm getting stale. In any case, while it's sort of comforting to know that there isn't any royal fuckup in my application packet that kept me from getting interviews, it's pretty frustrating not to have any idea how to prevent this from happening again. Shit, fuck.

It's also pretty frustrating because I really do feel like my career is going pretty well, notwithstanding my many failures on the job market. My teaching and publishing are going well. I more than satisfy the requirements for tenure at my current institution, and I have done this in well under the time allotted for tenure-line faculty here, and I have done this while managing a course load that is approximately double that of my tenure-line colleagues. I feel like I should be getting interviews.

I mean, I'm not trying to say that I "deserve" a job. 'Deserve' is a strong word, and makes it sound like there is an injustice here, as though I have been wronged. I don't think I have been wronged. And I don't teach at Princeton, and my institution's requirements for tenure are not Princeton's, and I wouldn't qualify for tenure at Princeton. I'm not a hot shot. But I would qualify for tenure (or be very close to it) at most of the places I sent applications to this fall, and I think that there's something kind of wrong with a situation where tenurable candidates aren't getting any interviews. And I particularly hate that it's me.

Now, look. I'm not trying to be all complainy. I've said before that, except for the job market, my career is going pretty well. It is. Except for the job market, my whole life is going pretty well. I've got a good job where I work hard at work that I find meaningful and where my hard work is appreciated. I get along with my colleagues really well. I have a modicum of year-to-year job security--more than VAP positions ordinarily come with--and this makes an enormous, incalculable difference in my life. I'm really happy with the philosophy I'm producing, and with my publication record. I literally get paid to talk about philosophy. My marriage is good, and my kid is healthy and happy. When people ask me how I'm doing, I say, "good," because I am.

But I sure do fucking hate having to be on the fucking job market every god damn year.

--Mr. Zero

San Jose State University calls out Sandel and MOOCs

Awesomeness ensues.

http://chronicle.com/article/The-Document-an-Open-Letter/138937/?cid=at&utm_source=at&utm_medium=en

~zombie