Submissions get matched to reviewers through several means, outlined here.
As an initial reminder, it is your choice to use any or all of the steps below. You can choose to do none of them and you will probably still be assigned perfectly reasonable papers. Some people enjoy having more control over what they are assigned, and others prefer to spend their time in other ways.
1. Your Reviewing Preferences
You can choose to fill out the review preferences survey we circulated via email. In the survey you can choose up to three areas for which you wanted to review, choose whether to review short paper submissions, long paper submissions or both, and select a reduced review load.
We understand researchers are often interdisciplinary, so except for a few areas with many reviewers, we have assigned you to your selected areas. This allows you to look at fewer papers while bidding (step 2).
If you chose not to communicate your review preferences, then you are looking at lots of papers right now – whatever works for you!
2. Bidding
You can choose to bid on papers in START (also known as softconf; https://www.softconf.com/naacl2018/papers). If you bid, these selections are taken into account by the area chairs when assigning reviewers.
How to bid? When you log in at the NAACL 2018 START website, you will see a user console with one or more tracks (if you specified areas in step 1). Click on “Track Committee Console” and then “Submission Selection Form” in one or more of the areas to bid. NB only bid for long paper submission tracks now; if you are also (or only) reviewing short paper submissions, bidding will be open after the short paper submission deadline.
The area chairs also look at your overall reviewing load across areas to keep it under your chosen load. As a final check, we will look at the overall reviewing load per reviewer on December 31st for long paper submissions, and January 25th for short paper submissions.
If you opted to review both long and short paper submissions, you may choose to bid on each type of submission. The system will keep track of your overall reviewing load.
3. Toronto Paper Matching System Scores
The area chairs will look at your Toronto Paper Matching System scores for the submissions they are tracking, and use this information along with bids and their own expert knowledge to assign reviewers.
We had hoped that the submissions would be sorted by TPMS score while you are bidding but sadly, that’s not yet possible. However, your TPMS profile once set up can be used again and again, so for future *ACL conferences you won’t have to go through this step and we hope this functionality will be in place for them. If you choose to check your top 5 matched papers in an area, we will have a service set up tomorrow.
Timeline
As a reminder, the timeline for review of long paper submissions is:
Dec 15 Long paper submissions due
Dec 16 [Program chairs] assign long paper submissions to areas
Dec 17-Dec 23 [Reviewers] long paper submission bidding
Dec 18-Dec 30 [Area chairs] long paper submission pre-review checks; reviewer assignments
Jan 1-Jan 24 [Reviewers] long paper submission reviews
Jan 18-Jan 24 [Area chairs] hunt down late or incomplete reviews
Jan 25-Feb 1 [Authors] review response period
Feb 2-Feb 9 [Reviewers] review author responses
Feb 2-Feb 9 [Area chairs] make accept/reject suggestions for long paper submissions
Feb 10-Feb 14 [Program chairs] make final accept/reject decisions for long paper submissions
And for short papers is:
Jan 10 Short paper submissions due
Jan 11 [Program chairs] assign short paper submissions to areas
Jan 12-Jan 18 [Reviewers] short paper submission bidding
Jan 13-Jan 25 [Area chairs] short paper submission pre-review checks; reviewer assignments
Jan 26-Feb 20 [Reviewers] short paper submission reviews
Feb 11-Feb 22 [Area chairs] hunt down late or incomplete reviews; make accept/reject suggestions for short paper submissions
Feb 23-27 [Program chairs] make final accept/reject decisions for short paper submissions
NAACL HLT 2018 Chairs Blog 