By the long paper review deadline January 25, which is also the start date of the author response period, we were still missing 63 reviews for long papers. The PC chairs and area chairs had to kick-off a painful chasing game:
- keep sending reminder emails to late reviewers, if no response within one day then:
- recruit one or two ad-hoc reviewers for each paper that is missing reviews. In total we recruited 30+ ad-hoc reviewers.
By 5pm ET February 2, one day before the author response deadline, we are only missing 3 reviews.
By 2pm ET February 3, the author response deadline, we are only missing 1 review. In case this final review arrives a little late after the deadline, we will figure out a way to allow the authors to see the review and update their responses within a few days.
A few papers got 4 or even 5 reviews due to the uncertainty at chasing and recruiting reviewers. We noticed that many late reviewers are senior researchers and never replied our reminder emails. On the other hand, many ad-hoc reviewers are also senior researchers who immediately said yes to our requests. A senior area chair even volunteered to review a paper that is a missing review in the last minute. We are very grateful for (and inspired by) their dedications.
So we can almost call this chasing game a victory. But it was a big pain and absolutely not fun for the PC chairs and area chairs. It’s not fair for authors of papers missing reviews because they had to keep waiting while facing a response deadline. It’s also not fair for ad-hoc reviewers. Although almost all of them are the trustees of PC chairs and area chairs, we still owe a BIG THANK-YOU to them for submitting reviews within such a short time frame, especially many of them are working on their own paper submissions for ACL and other deadlines. We will acknowledge all ad-hoc reviewers, together with the best reviewers, in our PC chair report at the conference.
In addition, the PC chairs and area chairs have also urged some reviewers to elaborate and refine their reviews to make them more constructive. We will continue to do so after receiving all author responses.
Our suggestions for the future:
- PC committee members (reviewers): if you somehow can predict that you would be too busy to provide constructive reviews, please say no to the PC invitation or notify the PC chairs early so they can modify review assignment. That would help the chairs A LOT.
- PC chairs and area chairs: maybe consider recruiting a small ad-hoc review committee in advance.