177 lines
5.2 KiB
Plaintext
177 lines
5.2 KiB
Plaintext
---
|
|
template: templates/talk_slides_v1.erb
|
|
title: How to Review a Paper
|
|
---
|
|
<section>
|
|
<h2>How to <s>Write</s> Review a Paper</h2>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Why do you care?</h3>
|
|
|
|
<ul>
|
|
<li>Your advisor will one day ask you to review a paper.</li>
|
|
<li class="fragment">You will graduate and one day be asked to review a paper.</li>
|
|
<li class="fragment">Your papers will be reviewed like this... get into your reviewer's head!</li>
|
|
</ul>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>My Process</h3>
|
|
|
|
<dl>
|
|
<dt>Make the paper scribblable</dt>
|
|
<dd>I load the paper into my tablet or print it out. It's important that (a) I'm comfortable, (b) distraction-free, and (c) can make unrestricted markings.</dd>
|
|
|
|
<dt>Read-Through</dt>
|
|
<dd>I generally read the paper in a single pass. But I write down <b>everything</b>.</dd>
|
|
|
|
<dt>Review notes</dt>
|
|
<dd>Back at a text editor/CMT/etc... I go through the notes and ensure that each note I made is (a) addressed in the paper, and/or (b) appears in my review.</dd>
|
|
</dl>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Criterion</h3>
|
|
|
|
<ul>
|
|
<li>Motivation</li>
|
|
<li>Completeness</li>
|
|
<li>Validity</li>
|
|
<li>Readability</li>
|
|
</ul>
|
|
|
|
<p class="fragment">A good paper doesn't need to win in all categories.</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Motivation</h3>
|
|
|
|
<p>Does the paper address a relevant problem or provide useful insights?</p>
|
|
|
|
<ul>
|
|
<li>The paper identifies a new problem.</li>
|
|
<li>The paper provides new insights on an old problem.</li>
|
|
<li>The paper identifies new applications of old techniques.</li>
|
|
</ul>
|
|
|
|
<p class="fragment">Why would someone want to read this paper?</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Completeness</h3>
|
|
|
|
<p>Does the paper solve the problem it set out to solve?</p>
|
|
|
|
<ul>
|
|
<li>Validate the list of claims (if present).</li>
|
|
<li>Does the paper have a broad/narrow enough scope?</li>
|
|
<li>Is the initial motivation addressed? <ul>
|
|
<li>... or has a sufficient milestone been met?</li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<p class="fragment">Is the paper clickbait?</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Validity</h3>
|
|
|
|
<p>Are statements in the paper correct?</p>
|
|
|
|
<ul style="font-size: 50%">
|
|
<li>System design/Algorithms <ul>
|
|
<li>Unexpected runtime costs.</li>
|
|
<li>Unhandled corner cases.</li>
|
|
</ul></li>
|
|
<li>Experiments <ul>
|
|
<li>Experimental design doesn't test what the authors claim.</li>
|
|
<li>Results don't agree with author's claims.</li>
|
|
</ul></li>
|
|
<li>Proofs <ul>
|
|
<li>Sanity check</li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<p class="fragment">How would you solve the problem the authors pose yourself?</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Readability</h3>
|
|
|
|
<p>Is the paper written clearly</p>
|
|
|
|
<ul>
|
|
<li>There is a clearly stated problem / objective.</li>
|
|
<li>All background topics not covered in [grad level class] are outlined.</li>
|
|
<li>System design and formalisms are clear, precise, and complete.</li>
|
|
<li>The paper is free of English bugs, grammar bugs, typos, etc...</li>
|
|
</ul>
|
|
|
|
<p class="fragment">Do you understand the paper?</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Reading the Paper: Milestones</h3>
|
|
|
|
<dl>
|
|
<dt>Introduction</dt>
|
|
<dd>Do I understand the problem the paper is solving?</dd>
|
|
<dd>How would I go about solving the problem the paper outlines?</dd>
|
|
<dd>How would I go about measuring a solution to this problem?</dd>
|
|
|
|
<dt>Background</dt>
|
|
<dd>Do I have a reasonable understanding of the techniques the paper plans to use?</dd>
|
|
</dl>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Reading the Paper: Milestones</h3>
|
|
|
|
<dl>
|
|
<dt>Algorithm/Data Structures</dt>
|
|
<dd>Do I understand the approach the authors are taking?</dd>
|
|
<dd>If the approach doesn't line up with my own, why?</dd>
|
|
|
|
<dt>Experiments</dt>
|
|
<dd>Are all of my expected experiments from earlier addressed?</dd>
|
|
<dd>Do the experiments measure what the authors want to measure?</dd>
|
|
<dd>Are the datasets reasonable/representative of the motivating workloads?</dd>
|
|
<dd>Do the graphs support the paper's claims?</dd>
|
|
</dl>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Feedback</h3>
|
|
|
|
<p>Be Specific</p>
|
|
|
|
<ul>
|
|
<li>Communicate to the authors at least one way to address your concern.</li>
|
|
<li>Establish a clear metric that can test whether the concern is addressed.</li>
|
|
<li>Include citations where possible.</li>
|
|
<li>Differentiate suggestions from criticism.</li>
|
|
<li>Refer to specific lines of text.</li>
|
|
</ul>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Receiving Feedback</h3>
|
|
|
|
<p>Why did the reviewer write it?</p>
|
|
|
|
<ul>
|
|
<li>Did the reviewer misunderstand what you wrote?
|
|
<div class="fragment">What can you do to make it clearer?</div></li>
|
|
<li>Did the reviewer ignore an important point?
|
|
<div class="fragment">What can you do to make it more visible?</div></li>
|
|
<li>Did the reviewer not have the right background?
|
|
<div class="fragment">What can you do to make it more accessible?</div></li>
|
|
<li>Did the reviewer disagree with the motivation?
|
|
<div class="fragment">"Pitch" the motivation to your fellow students/faculty.</div></li>
|
|
</ul>
|
|
</section>
|
|
|
|
<section>
|
|
<p>The reviewer is a sample of the people who will be reviewing your paper. They may be wrong, but you still need to communicate to others like them to get an accept!</p>
|
|
</section> |