As General Chair I’ve watched with interest the widely varying feedback we have gotten on our new review form (https://naacl2018.wordpress.com/2017/12/14/our-new-review-form/). We’ve tried to keep in mind that people in general might be resistant to change, as well as the program chairs’ motivations for designing a new form. So we have had feedback:
- From experienced reviewers who have been reviewing for a long time. Some of them dislike the new form; they’d rather have their old unstructured form back.
- From less experienced reviewers, and reviewers who advise students. This group tends to in general think the new form is very useful.
- From experienced reviewers who have been program chairs for ACL sponsored and other NLP conferences, as well as for AI conferences, NIPS etc. This group has applauded our new review form and the whole redesign of the reviewing process, perhaps because they personally have had those difficult nights after the program decisions were made, where they realized that which “grey area” papers got in, and which ones didn’t, was not as well-informed a decision as one might hope.
- While waiting to get the author rebuttals back, we have received a lot of comments that authors like the new review form, because it lets them construct better rebuttals. They can also easily see when the reviewer did not provide detailed feedback on strengths and weaknesses or contributions, and easily pick out places where it seems the reviewer misunderstood one of the main points of the paper.
The new review form is designed to be used by the area chairs to allow them to be more editorial in their selection of papers – in particular, to help with the vast middle of papers where it has traditionally been very unclear what to do. This year, along with the design of a new review form, the program chairs also reduced the load on each area chair, to allow them to spend more time on qualitative analysis of the papers. The new review form should give them much more detailed information than they have had available in the past. Please see https://naacl2018.wordpress.com/2018/01/31/its-not-all-about-the-numbers/ for information about the instructions to the area chairs and how they are supposed to proceed.
This is all in a context where we are all aware of how fast NLP is growing. People complain about review quality and reviewer overload. We can only fix reviewer
overload by having a more diverse reviewer pool – including “not the usual suspects”. That however risks further reducing review quality – unless we give “not the usual suspects” a more structured review form. One could argue about what that structure should look like, and in fact this year there will be three variants: ours, which asks for comment by contribution type, the ACL one, which asks for comments by strengths and weaknesses and questions, and the COLING one, which has different questions for different types of papers.
We’d love to hear from more authors. What do you think of the new review form in terms of helping you (1) write your rebuttal or (2) revise the paper for resubmission elsewhere or (3) prepare the final version of your paper for publication?
We are looking forward to assessing the impact of our new reviewing process on the quality of the conference and invite the ACL community to keep thinking about these issues as well and keep those cards and letters coming!
NAACL HLT 2018 Chairs Blog 