You may recall that one of our goals for NAACL HLT 2018 is to ensure broad participation. We recruited a pool of 1247 program committee members. To do this in a non-biased fashion, we re-used a method used for NAACL HLT 2016 – we invited people who have published repeatedly in *CL venues (ACL, NAACL, EMNLP, etc.). We thank Dragomir Radev for giving us a list of these people from the ACL Anthology, and START for giving us a list of reviewers and authors from previous *CL conferences so we could match names to START user ids.
We now report statistics on our reviewer pool. (We apologize for the heterogeneous nature of the graphics!) Then, we give some ideas for future program chairs who also want to ensure broad participation.
Paper Types
Most of our reviewers were willing to review long or short paper submissions.
Areas
We report here the areas for which respondents said they would be comfortable reviewing. There are some outliers with respect to submissions: Generation and Summarization got a record number of submissions, both absolute and as a percentage of all submissions, while NLP Applications did not get a vast number of submissions.
Load
In our reviewer invitation letter, we offered a max load of three submissions per reviewer. Most of our reviewers were happy to review three. 24% asked to review only two, and 6% offered to review more than three.
Level of Experience
Most respondents are 0-4 years post PhD. This is a very young reviewer pool (although some more senior program committee members did not fill out the survey).
We tried to only invite to the program committee people with PhDs, with significant experience in the field, or within 1 year of finishing a PhD. Junior graduate students may be reviewers – and in fact reviewing under the supervision of an advisor can be a terrific educational opportunity – but we think that graduate students should not, in general, be program committee members. For one thing, they should be developing their own research skills, and for another, it is known that more junior researchers are more harsh in their reviews.
Affiliation
34% of respondents work in industry; 13% are post-docs; 46% are faculty, 6% work in government labs, 2% are consultants, and the remainder have other types of position (including student). Some people listed multiple affiliation types.
Gender
78% of respondents are male. This is a slightly higher percentage than the percentage of ACL attendees (ref).
Geography
Most respondents are from North America and Europe (there are two parts to the “Europe” responses owing to some disagreement about the UK ;)).
Recommendations for Future Program Chairs
NAACL in particular should do a better job of involving researchers from Central / South America. We had difficulty identifying potential area chairs and program committee members from these regions.
We recommend future program chairs to continue to track and ideally, report other aspects of diversity (industry/academia, gender, specific institutions that have many researchers). If the numbers look very off during the program committee selection process, changes may be made to ensure broader representation.
Also, we recommend future program chairs to recruit a broad pool using publication and citation information, rather than relying on “who they know” – for one thing, very few of us know enough people to construct a program committee for the size of conference we are now dealing with.
Future program chairs should have a policy about recruiting junior and senior graduate students as program committee members.
Future program chairs may wish to think about which areas are “trending” and recruit more reviewers from those areas, including going to the SIGs.
As a matter of tactics, we were not aware that START allows program chairs to ask reviewers additional “profile” questions within the system. Future program chairs may wish to use START instead of an external survey mechanism for collecting this type of data.
We will share with future program chairs the list of reviewers who indicated they would be willing to review for a future *CL conference (as well as those who indicated they would not). We recommend future program chairs to collect this information and pass it on in their turn.
NAACL HLT 2018 Chairs Blog 