Some reviewers and authors have contacted us to inquire if the only factor in the eventual decision on a submission will be the overall score. That’s not how the review process should work, and it is not how it will work for NAACL HLT 2018.
We know (http://inverseprobability.com/2014/12/16/the-nips-experiment) that reviewing is imperfect. We constructed a review form designed to get reviewers to profile the papers and to advocate for/against in various ways. This will allow area chairs to assess any reviewer disagreement more explicitly. And authors should be able to understand from the review comments how they can improve their submissions for eventual publication either at our venue or elsewhere.
Area chairs, in turn, have the responsibility and the right to advocate for/against submissions based on the submission itself, the reviews, and what they observe about the whole slate of submissions within their view. They will be working together in groups of 2 or 3 to make accept/reject suggestions, and will write meta-reviews for any submission that is not a clear accept or a clear reject. Each area chair is handling no more than 30 submissions (across both long and short), so each area chair has ample time to really investigate the submission and make an informed accept/reject suggestion.
The instructions for area chairs say:
- Look at all the reviews for a submission – if there is violent disagreement, you may ask the reviewers to discuss, but should not attempt to force a consensus.
- Consider the scores and comments for replicability, soundness/correctness, substance and originality.
- Think about the contributions the paper makes.
-
If the reviews for a paper note mishandling of data or very poor readability, please think carefully before recommending this paper for acceptance.
- No matter what you do, please look beyond the Overall_Score.
- Try to recommend a portfolio of papers that, across the whole portfolio, demonstrates diversity in contributions.
The program chairs will use the area chairs’ suggestions and metareviews, along with our own consideration of the reviews, to make final accept/reject decisions.
tl;dr The overall score should not be the focus of reviewers’, authors’ and area chairs’ discussions – the comments should be.
NAACL HLT 2018 Chairs Blog 