By the long paper review deadline January 25, which is also the start date of the author response period, we were still missing 63 reviews for long papers. The PC chairs and area chairs had to kick-off a painful chasing game:
- keep sending reminder emails to late reviewers, if no response within one day then:
- recruit one or two ad-hoc reviewers for each paper that is missing reviews. In total we recruited 30+ ad-hoc reviewers.
By 5pm ET February 2, one day before the author response deadline, we are only missing 3 reviews.
By 2pm ET February 3, the author response deadline, we are only missing 1 review. In case this final review arrives a little late after the deadline, we will figure out a way to allow the authors to see the review and update their responses within a few days.
A few papers got 4 or even 5 reviews due to the uncertainty at chasing and recruiting reviewers. We noticed that many late reviewers are senior researchers and never replied our reminder emails. On the other hand, many ad-hoc reviewers are also senior researchers who immediately said yes to our requests. A senior area chair even volunteered to review a paper that is a missing review in the last minute. We are very grateful for (and inspired by) their dedications.
So we can almost call this chasing game a victory. But it was a big pain and absolutely not fun for the PC chairs and area chairs. It’s not fair for authors of papers missing reviews because they had to keep waiting while facing a response deadline. It’s also not fair for ad-hoc reviewers. Although almost all of them are the trustees of PC chairs and area chairs, we still owe a BIG THANK-YOU to them for submitting reviews within such a short time frame, especially many of them are working on their own paper submissions for ACL and other deadlines. We will acknowledge all ad-hoc reviewers, together with the best reviewers, in our PC chair report at the conference.
In addition, the PC chairs and area chairs have also urged some reviewers to elaborate and refine their reviews to make them more constructive. We will continue to do so after receiving all author responses.
Our suggestions for the future:
- PC committee members (reviewers): if you somehow can predict that you would be too busy to provide constructive reviews, please say no to the PC invitation or notify the PC chairs early so they can modify review assignment. That would help the chairs A LOT.
- PC chairs and area chairs: maybe consider recruiting a small ad-hoc review committee in advance.
NAACL HLT 2018 Chairs Blog 
This always, always happens without fail for every conference. Realistically, it’s unavoidable given the numbers of people involved: people’s kids get sick, or their employers overload them with work at unpredictable times, or (yes) they overcommit, or whatever. You as the chair have no control over these things, and the only sane strategy is to explicitly plan a triage phase, which mainly has two elements:
(1) I strongly second your recommendation to recruit reliable emergency backup reviewers in advance (i.e. when you do initial recruiting of reviewers) and tell them that they *will* be on call to turn around a few reviews in a 48-hour window after the official review deadline. Given the current size of the community, it’s basically impossible to know all of the reviewers involved, but this much smaller pool of reliable reviewers should just be people who you know you can rely on to get the job done.
(2) The other thing that I strongly advocate is to include a 2- or 3-day window between the review deadline and the beginning of the author response period. I know, the schedule is already tight, but this really just comes down to shortening the review period and author response period by a day apiece. This window is when you put the emergency reviewers into action, prod the others like crazy, and do some reading yourself (if you’ve got time).
When I’ve used both of these strategies in tandem, every single paper had at least three reviews by the time the authors saw them. (I admit, for about an order of magnitude fewer papers… which still entailed a few hundred individual reviews.) There is some stress during the triage phase, but when you’ve planned for it and you know you’ve got a dedicated team of reliable people working on it, it is not too bad. It’s far less bad then the stress radiating from the panicked emails filling up your inbox from anxious authors (often students) who are waiting on pins and needles for a missing review that will make their paper sink or swim. There will still be lots of angry authors for other reasons… but that’s probably unavoidable.
LikeLike
Thank you so much!
LikeLike