Our New Review Form

In this post, we will introduce all of you to our new review form for the NAACL HLT 2018 research track. This review form was created in cooperation with ACL 2018; their final review form is a little different, but substantially similar.

Why a New Review Form?

The new review form is in service of our goals to manage the integrity and quality of the publication process and to construct a great program.

To motivate our new review form, let’s think about who the review form is for:

• The reviewer – the reviewer, after all, completes the review form. Many of the items in our review form are designed to guide the reviewer through an assessment of the submission in a structured fashion.
• The area chair and program chairs – the area chair and program chairs use the completed review to make accept/reject decisions. Our review form gives a multi-faceted view of submissions, allowing area chairs and program chairs to assess different types of contribution presented in a paper  and their likely impact.
• The author – the completed review is presented to the author, who may use it to improve their paper, either to create a final publication or to resubmit to another venue. Our hope is that the more structured feedback will be helpful to authors.

Sections of the New Review Form

We address the rest of this post primarily to reviewers.

Instructions and Consent to Use Reviews for Research

The first section of the new review form reminds you of the ACL review guidelines and standard confidentiality rules for reviewing. Also, you are asked to consent for your (anonymized) review to be included in a new review corpus to be jointly created by the NAACL HLT and ACL 2018 program chairs, and released to the NLP community no sooner than 6 months after the conference (this timeline is being negotiated between NAACL HLT and ACL program chairs). Authors will also be invited to consent, during the author response period. Reviews will be included in the corpus only on consent of both authors and reviewers.

Appropriateness

You should assess whether the submission is appropriate to NAACL HLT 2018 and to the track to which it was submitted. If it is not, provide a justification and notify the area chair before proceeding.

Contributions

This section forms the biggest part of the review form. As we outlined in an earlier blog post, we want to identify the types of contribution described in a submission, as well as the strengths and weaknesses of each type of contribution.

You will assess whether, in your opinion as the reviewer (not the opinion of the author), the submission describes novel contributions of the following types:

• New NLP tasks / applications
• New methods
• New theoretical, algorithmic or empirical results
• New data / resources
• New systems / software
• New evaluation metrics / methods
• Other contributions

Any one submission will probably only have contributions of one or two types. For each type of contribution you think the submission describes, you will briefly list the strengths and weaknesses of the contribution, and assess its potential impact on the field.

For example, for empirical results the review form asks:

1. Do the contributions of the submission include new empirical results?

   An empirical result may include a corpus study, evaluation or controlled experiment done to test a hypothesis. For submissions presenting empirical results, the authors should describe the hypothesis being tested and adequately account for confounds.

2. If the work includes new empirical results (such as evaluations, experiments or corpus analyses; this includes interesting negative results), provide a brief description of the hypothesis/es tested, and their novelty and substance:

3. If the work includes new empirical results, provide a brief description of the method for testing the hypothesis/es, and analysis of strengths and weaknesses (e.g. confounds, lack of error analysis):

4. How interesting and impactful might these new empirical results be to the NAACL HLT community? (Very, Somewhat, Not at all, N/A)

For software/systems, the review form asks:

1. Do the contributions of this work include new software / systems?

   The software or system need not necessarily be provided with the submission – for example, if it is proprietary – although to the extent possible, researchers are encouraged to share software and systems in the interests of reproducible science.

2. If the contributions of the work include new software, provide a brief description and analysis of strengths and weaknesses:

3. How useful might the new software/systems be be to the NAACL HLT community? (Very, Somewhat, Not at all, N/A)

Handling of Data, Resources and Human Participants

You will assess whether the submission adheres to the NAACL HLT 2018 guidelines regarding handling of data, resources and human participants.

Related Work

You will evaluate whether the submission adheres to the ACL guidelines for authors, and whether the discussion of related and prior work motivates and supports the main claims of the submission in an appropriate scholarly manner.

In particular, the review form asks:

1. Does the discussion of related and prior work motivate and support the main claims of the submission in an appropriate scholarly manner? Is it complete?

   If you feel references are incomplete, be sure to include the relevant references in your comments.

   • 5 = Comparison to prior work is superbly carried out given the space constraints.

   • 4 = Comparisons are mostly solid, but there are some missing references.

   • 3 = Comparisons are weak, very hard to determine how it compares to previous work.

   • 2 = Only partial awareness or understanding of related work, or a flawed empirical comparison.

   • 1 = Little awareness of related work, or lacks necessary empirical comparison.

2. Does the discussion of related and prior work adhere to the ACL author guidelines on citation (https://www.aclweb.org/adminwiki/index.php?title=ACL_Author_Guidelines)?

3. Provide any comments regarding the discussion of related and prior work:

Readability, Style and Format

You will evaluate the extent to which the paper is clear and well written, and whether it adheres to the NAACL HLT 2018 style and format guidelines.

ACL Author Guidelines; Facilitating Double Blind Review

You will evaluate whether the paper adheres to the ACL guidelines for authors regarding facilitation of double blind review. If you think you can identify the authors, you will be asked to indicate how, so that we can evaluate the effectiveness of our double blind review process.

Final Comments

You will be asked to provide an overall score for the submission:

1. Based on your review of this submission, should it be accepted to the NAACL HLT 2018 research track? In deciding on your ultimate recommendation, please think over all your responses above. We want a conference full of creative, original, sound and timely work. Prefer work that is inventive and will stimulate new approaches over work that is solid but incremental. Remember also that the author has about a month to address reviewer comments before the camera-ready deadline.

   • 6 = Transformative: This paper is likely to change our field.

   • 5 = Exciting: The work presented in this submission includes original, creative contributions, the methods are solid, and the paper is well written.

   • 4 = Interesting: The work described in this submission is original and basically sound, but there are a few problems with the method or paper.

   • 3 = Uninspiring: The work in this submission lacks creativity or originality. I’m ambivalent about this one.

   • 2 = Borderline: This submission has some merits but there are significant issues with respect to originality, soundness, replicability or substance, readability, etc.

   • 1 = Poor: I cannot find any reason for this submission to be accepted.

In making accept/reject suggestions, area chairs will look at the matrix of contributions made by different submissions as well as appropriateness of the work to the field, readability of the paper and completeness of discussion of related work in each submission. So the reviews will be useful in making an interesting, varied assortment of papers for the conference.

We encourage all reviewers to take a look at the review form before starting to read their assigned submissions as it will help in reading the papers.

As in previous years, we will have an author response period. However, we will only have this for long paper submissions as with this type of submission there may be more ambiguity.