No information available yet.
FAQ on Double-Blind Reviewing
FAQ on Double-Blind Reviewing
General
Q: Why are you using double-blind reviewing?
A: Studies have shown that a reviewer’s attitude toward a submission may be affected, even unconsciously, by the identity of the authors. We want reviewers to be able to approach each submission without any such, possibly involuntary, pre-judgment. Many computer science publications have embraced double-blind reviewing. PLDI has used it for several years now and doing so is stipulated in the Practices of PLDI (under update).
Q: Do you really think blinding actually works? I suspect reviewers can often guess who the authors are anyway.
A: It is rare for authorship to be guessed correctly, even by expert reviewers, as detailed in this study.
Q: Couldn’t blind submission create an injustice where a paper is inappropriately rejected based upon supposedly-prior work which was actually by the same authors and not previously published?
A: Reviewers are held accountable for their positions and are required to identify any supposed prior work that they believe undermines the novelty of the paper. Any assertion that “this has been done before” by reviewers should be supported with concrete information. The author response mechanism exists in part to hold reviewers accountable for claims that may be incorrect.
For authors
Q: What exactly do I have to do to anonymize my paper?
A: Use common sense. Your job is not to make your identity undiscoverable but simply to make it possible for reviewers to evaluate your submission without having to know who you are. The specific guidelines stated in the call for papers are simple: omit authors’ names from your title page, and when you cite your own work, refer to it in the third person. For example, if your name is Smith and you have worked on amphibious type systems, instead of saying “We extend our earlier work on statically typed toads [Smith 2004],” you might say “We extend Smith’s [2004] earlier work on statically typed toads.” Also, be sure not to include any acknowledgements that would give away your identity. In general, you should aim to reduce the risk of accidental unblinding. For example, if your paper is the first to describe a system with a well-known name or codename, or you use a personally-identifiable naming convention for your work, then use a different name for your submission (which you may indicate has been changed for the purposes of double-blind reviewing). You should also avoid revealing the institutional affiliation of authors or at which the work was performed.
Q: I would like to provide supplementary material for consideration, e.g., the code of my implementation or proofs of theorems. How do I do this?
A (and also see the next question): On the submission site there will be an option to submit supplementary material along with your main paper. This supplementary material should also be anonymized; it may be viewed by reviewers during the review period, so it should adhere to the same double-blind guidelines.
Q: My submission is based on code available in a public repository. How do I deal with this?
A: Making your code publicly available is not incompatible with double-blind reviewing. You should do the following. First, cite the code in your paper, but remove the actual URL and, instead say “link to repository removed for double-blind review” or similar. Second, if, when writing your author response, you believe reviewer access to your code would help, say so in your author response (without providing the URL), and upload a zip file containing the code under supplemental materials (but make sure that the code/documentation does not reveal the identity of the authors).
Q: I am building on my own past work on the WizWoz system. Do I need to rename this system in my paper for purposes of anonymity, so as to remove the implied connection between my authorship of past work on this system and my present submission?
A: Maybe. The core question is really whether the system is one that, once identified, automatically identifies the author(s) and/or the institution. If the system is widely available, and especially if it has a substantial body of contributors and has been out for a while, then these conditions may not hold (e.g., LLVM or HotSpot), because there would be considerable doubt about authorship. By contrast, a paper on a modification to a proprietary system (e.g., Visual C++, or a research project that has not open-sourced its code) implicitly reveals the identity of the authors or their institution. If naming your system essentially reveals your identity (or institution), then anonymize it. In your submission, point out that the system name has been anonymized. If you have any doubts, please contact the Associate Editor.
Q: I am submitting a paper that extends my own work that previously appeared at a workshop. Should I anonymize any reference to that prior work?
A: No. But we recommend you do not use the same title for your submission, so that it is clearly distinguished from the prior paper. In general, there is rarely a good reason to anonymize a citation. One possibility is for work that is tightly related to the present submission and is also under review. When in doubt, contact the Associate Editor.
Q: Am I allowed to post my (non-blinded) paper on my web page? Can I advertise the unblinded version of my paper on mailing lists or send it to colleagues? Can I give a talk about my work while it is under review? How do I handle social media? What about arXiv?
A: We have developed guidelines, described here, to help everyone navigate in the same way the tension between the normal communication of scientific results, which double-blind reviewing should not impede, and actions that essentially force potential reviewers to learn the identity of the authors for a submission. Roughly speaking, you may (of course!) discuss work under submission, but you should not broadly advertise your work through media that is likely to reach your reviewers. We acknowledge there are gray areas and trade-offs; we cannot describe every possible scenario.
Things you may do:
- Put your submission on your home page.
- Discuss your work with anyone who is not on the Review Committee, or with people on the committees with whom you already have a conflict.
- Present your work at professional meetings, job interviews, etc.
- Submit work previously discussed at an informal workshop, previously posted on arXiv or a similar site, previously submitted to a conference not using double-blind reviewing, etc.
Things you should not do:
- Contact members of the Review Committee about your work, or deliberately present your work where you expect them to be.
- Publicize your work on major mailing lists used by the community (because potential reviewers likely read these lists).
- Publicize your work on social media if wide public [re-]propagation is common (e.g., Twitter) and therefore likely to reach potential reviewers. For example, on Facebook, a post with a broad privacy setting (public or all friends) saying, “Whew, PLDI paper in, time to sleep” is okay, but one describing the work or giving its title is not appropriate. Alternatively, a post to a group including only the colleagues at your institution is fine. Reviewers will not be asked to recuse themselves from reviewing your paper unless they feel you have gone out of your way to advertise your authorship information to them. If you are unsure about what constitutes “going out of your way”, please contact the Associate Editor.
Q: Will the fact that PLDI is double-blind have an impact on handling conflicts-of-interest?
A: Double-blind reviewing does not change the principle that reviewers should not review papers with which they have a conflict of interest, even if they do not immediately know who the authors are. Authors declare conflicts-of-interest when submitting their papers using the guidelines in the call-for-papers. Papers will not be assigned to reviewers who have a conflict.
For reviewers
Q: What should I do if I learn the authors’ identity? What should I do if a prospective author contacts me and asks to visit my institution?
A: If you feel that the authors’ actions are largely aimed at ensuring that potential reviewers know their identity, contact the Associate Editor. Otherwise, you should not treat double-blind reviewing differently from other reviewing. In particular, refrain from seeking out information on the authors’ identity, but if you discover it accidentally this will not automatically disqualify you as a reviewer. Use your best judgment.
Q: If I am assigned a paper for which I feel I am not an expert, how do I seek an outside review?
A: PC members should write their own reviews and not delegate them to someone else. If doing so is problematic for some papers (e.g., you do not feel completely qualified), then please take the following steps: First, submit a review for your paper that is as careful as possible, outlining areas where you think your knowledge is lacking. Assuming we have sufficient expert reviews, that could be the end of it: non-expert reviews are valuable too, since conference attendees are by-and-large not experts for any given paper. Second, the review form provides a mechanism for suggesting additional expert reviewers to the PC Chair, who may contact them if additional expertise is needed. Please do not contact outside reviewers yourself.
Q: How do we handle potential conflicts of interest since I cannot see the author names?
A: The conference review system will ask that you identify conflicts of interest when you get an account on the submission system. Feel free to also identify additional authors whose papers you feel you could not review fairly for reasons other than those given (e.g., strong personal friendship).
Q: How should I avoid learning the authors’ identity if I am using web-search in the process of performing my review?
A: You should make a good-faith effort not to find the authors’ identity during the review period, but if you inadvertently do so, this does not disqualify you from reviewing the paper. As part of the good-faith effort, do not use search engines with terms like the paper’s title or the name of a new system being discussed. If you need to search for related work you believe exists, do so after completing a preliminary review of the paper.
Q: When will author identities be revealed?
A: The Review Committee will only learn the identities of authors of accepted papers following the second round of reviewing. The authors of rejected papers will remain anonymous to everyone except the Associate Editor.