Table of Contents
Notes on Good Reviewing Practices
Alexandra Meliou, Tim Kraska, and Sam Madden
(Liberally stolen from the SIGMOD 2016 reviewer guidelines)
Reviewing is a great and much appreciated service. When you prepare your reviews, please consider that they serve three purposes:
- To communicate your thoughts about the paper to the other members of the PC.
- To provide authors with feedback on how to improve their paper and proceed with their work, regardless of the paper decision.
- To justify and document final decisions.
Regardless of whether you vote to accept or reject a particular paper, we expect you to provide thoughtful and constructive feedback. We urge you to observe the following guidelines in your reviews:
1. Review length.
A good review should cover at least the following aspects of the paper in detail: problem focus and motivation, technical contributions, experimental evaluation, related work, and overall presentation. You should elaborate on all these points in the detailed comments section of the review form. We expect an ideal review to span at least 4-5 paragraphs, discussing these aspects. A very short review is very likely a bad review. Recall that one of the purposes of your review is to give authors constructive feedback, and a short review does not accomplish that.
Examples to avoid:
"Detailed comments: Please see strengths and weaknesses"
It is called "detailed comments" for a reason (in fact this year we are calling this section "Paper Review"!). Please discuss your feedback in detail, and provide constructive comments to the authors. The authors expect not only a clearly justified evaluation, but also specific and constructive advice on improving their paper.
2. Critiquing the paper's motivation and focus.
A good review should evaluate the importance and relevance of the problem that the authors list. The authors may not agree with your conclusion, but it is vital that they understand your rationale. For this reason, criticisms on the motivation of the problem should be clearly explained and justified.
Examples to avoid:
"The motivation of the paper is weak" It is not clear to the authors whether you were dissatisfied with their presentation or with the problem they focus on.
"This problem is not interesting" "I don't think anyone will use this"
Avoid basing your critique on personal opinions and biases. Perhaps the paper's focus is not interesting to you, but that does not mean that it is not important. Give specific arguments to explain why you believe it is not practical. It is also not sufficient to simply state that you didn't find the problem motivation convincing. You should explain why you believe the paper's focus is unrealistic / too narrow / too broad / no longer relevant.
More constructive:
"The introduction tries to establish the problem motivation by citing prior work, but the paper should stand more on its own. The motivation would improve by discussing real-world examples of this problem and the challenges of current technology."
3. Critiquing the technical contributions.
A big part of your review is to evaluate the technical contributions of the work. List the claims made in the paper, judge if they are applicable to the problem, and evaluate their expected importance, applicability, and usefulness. If you believe that some assumptions of the paper are too restrictive and make the problem unrealistic or narrow, explain that. As a reviewer, we rely on your expertise to evaluate the novelty of a paper's contributions. However, you should avoid tainting your evaluation with personal opinions. Judge the work that the authors are presenting, rather than the work that you have preferred to see.
Examples to avoid:
"The paper makes strong theoretical contributions in this domain, but it doesn't test them in a real-world, large-scale deployment" "Algorithm A is really interesting and insightful, but what about algorithm B that I just came up with?" "The paper solves the problem for 4 dimensions, but what about 5 dimensions?"
It is great to bring up additional contributions that you would like to see and ideas that could extend the presented work. But write them as suggestions, rather than a basis of criticism. Your review should judge the contributions that the paper makes, rather than the contributions it does not make. As a rule of thumb, if your criticism could apply to any paper (e.g., "the paper does A very well, but what about B") then it is not a fair criticism.
More constructive:
"Algorithm A is non-trivial and has low time complexity. However, it requires maintaining a large data structure in memory, which makes it impractical for large datasets."
4. Critiquing the evaluation.
Your review should evaluate whether the experiments validate the claims of the paper. If the paper claims that the proposed algorithm improves on the state-of-the-art, discuss whether the experiments compare the proposed approach with the appropriate methods, and against all expected metrics (e.g., efficiency, accuracy, scalability). Your review should note whether there are any inconsistencies in the results and whether there are observations that are not explained, justified, or discussed. We also rely on you to judge whether the experimental methods and datasets are appropriate, or if they introduce biases in the results.
Examples to avoid:
"The experiments compare the proposed approach with methods A, B, and C, but method D could also be applicable if one made these parameters flexible"
This is more often than not an unfair criticism. If a method needs non-trivial adjustments to apply in this work, and the paper already compares with suitable techniques, this should not be a basis of criticism.
"Larger-scale experiments are needed"
This comment by itself is not sufficient criticism. Communicate to the author what scale of experiments you would consider satisfactory.
More constructive:
"The experiments do not include comparisons with other methods, because this is the first work that addresses problem X. However, I believe that Algorithm ABC for problem Y could be applicable here with the following modifications:..."
5. Critiquing the writing and presentation.
You may often find the presentation of a paper unsatisfactory. It is good to list points of confusion, repetitiveness, poor structure, confusing notation, and lack of formalism. However, make sure that you provide concrete examples and pointers for your criticisms. For example, explain what about the notation confused you, which terms were not properly defined, and which statements should be made more rigorous. Communicating vague discontent about the readability of the paper is not constructive, because it doesn't explain to the authors how they can improve it.
Examples to avoid:
"The paper is unreadable"
Only OK if the paper is actually written in a foreign language. Otherwise, give some examples of the readability issues that you experienced.
"The notation is confusing"
Be more explicit about what you found problematic, and what you think might help.
More constructive:
"I found section 4.1 hard to follow. I believe that it would help if the authors moved the discussion at the end of section 4.3 to the beginning of section 4.1, as it gives the intuition behind the definitions"
"There is a clash of notation: sometimes \alpha is used to represent a query answer, but it is also the approximation factor in Algorithm 1."
6. Critiquing the related work.
You should ensure that the paper includes sufficient discussion of the related work, explains similarities and differences, and does not omit obvious connections. Keep in mind however that, unless you are reviewing a survey paper, there will always be some citations that could have been added but weren't. Do not criticize a paper for the omission of a particular citation if it already includes sufficient discussion of the related work in the particular area. Do point out though glaring omissions and important connections that were missed.
Examples to avoid:
"This work seems very similar to [X]"
This claim is vague and unsubstantiated. Provide clear discussion on whether particular claims have been demonstrated before in prior work.
"There is a large body of related work on problem X that the paper does not discuss"
You should make such a criticism more concrete by including particular references that you expected to see discussed. Suggest to the authors a few specific citations that they should include.
More constructive:
"The authors mention connections to [X] in the related work, but they do not explain why [X] is not directly applicable and why a new approach is needed."
7. Professionalism.
Be thoughtful and professional. Refrain from using language that is demeaning and ensure that your criticisms are constructive and clearly supported. Keep a friendly tone and avoid being condescending. A thoughtful and courteous review communicates to the authors that you read their paper with a positive attitude, and not with a mind to find faults with it. Well-structured arguments in your reviews help assure the authors that their papers were treated fairly and professionally. Remember that you are acting as a representative of the entire SIGMOD community, and thoughtful, constructive reviews are important in maintaining the integrity of our area.