140 lines
3.9 KiB
Plaintext
140 lines
3.9 KiB
Plaintext
---
|
|
template: templates/talk_slides_v1.erb
|
|
title: How to Review a Paper
|
|
---
|
|
<section>
|
|
<h2>How to Review a Paper</h2>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>My Process</h3>
|
|
|
|
<dl>
|
|
<dt>Make the paper scribblable</dt>
|
|
<dd>I load the paper into my tablet or print it out. It's important that (a) I'm comfortable, (b) distraction-free, and (c) can make unrestricted markings.</dd>
|
|
|
|
<dt>Read-Through</dt>
|
|
<dd>I generally read the paper in a single pass. But I write down <b>everything</b>.</dd>
|
|
|
|
<dt>Review notes</dt>
|
|
<dd>Back at a text editor/CMT/etc... I go through the notes and ensure that each note I made is (a) addressed in the paper, and/or (b) appears in my review.</dd>
|
|
</dl>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Criterion</h3>
|
|
|
|
<ul>
|
|
<li>Motivation</li>
|
|
<li>Completeness</li>
|
|
<li>Validity</li>
|
|
<li>Readability</li>
|
|
</ul>
|
|
|
|
<p>A good paper doesn't need to win in all categories.</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Motivation</h3>
|
|
|
|
<p>Does the paper address a relevant problem or provide useful insights?</p>
|
|
|
|
<ul>
|
|
<li>The paper identifies a new problem.</li>
|
|
<li>The paper provides new insights on an old problem.</li>
|
|
<li>The paper identifies new applications of old techniques.</li>
|
|
</ul>
|
|
|
|
<p>Why would someone want to read this paper?</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Completeness</h3>
|
|
|
|
<p>Does the paper solve the problem it set out to solve?</p>
|
|
|
|
<ul>
|
|
<li>Validate the list of claims (if present).</li>
|
|
<li>Does the paper have a broad/narrow enough scope?</li>
|
|
<li>Is the initial motivation addressed? <ul>
|
|
<li>... or has a sufficient milestone been met?</li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<p>Is the paper clickbait?</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Validity</h3>
|
|
|
|
<p>Are statements in the paper correct?</p>
|
|
|
|
<ul>
|
|
<li>System design/Algorithms <ul>
|
|
<li>Unexpected runtime costs.</li>
|
|
<li>Unhandled corner cases.</li>
|
|
</ul></li>
|
|
<li>Experiments <ul>
|
|
<li>Experimental design doesn't test what the authors claim.</li>
|
|
<li>Results don't agree with author's claims.</li>
|
|
</ul></li>
|
|
<li>Proofs <ul>
|
|
<li>Sanity check</li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<p>How would you solve the problem the authors pose yourself?</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Readability</h3>
|
|
|
|
<p>Is the paper written clearly</p>
|
|
|
|
<ul>
|
|
<li>There is a clearly stated problem / objective.</li>
|
|
<li>All background topics not covered in [grad level class] are outlined.</li>
|
|
<li>System design and formalisms are clear, precise, and complete.</li>
|
|
<li>The paper is free of English bugs, grammar bugs, typos, etc...</li>
|
|
</ul>
|
|
|
|
<p>Do you understand the paper?</p>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Reading the Paper: Milestones</h3>
|
|
|
|
<dl>
|
|
<dt>Introduction</dt>
|
|
<dd>Do I understand the problem the paper is solving?</dd>
|
|
<dd>How would I go about solving the problem the paper outlines?</dd>
|
|
<dd>How would I go about measuring a solution to this problem?</dd>
|
|
|
|
<dt>Background</dt>
|
|
<dd>Do I have a reasonable understanding of the techniques the paper plans to use?</dd>
|
|
|
|
<dt>Algorithm/Data Structures</dt>
|
|
<dd>Do I understand the approach the authors are taking?</dd>
|
|
<dd>If the approach doesn't line up with my own, why?</dd>
|
|
|
|
<dt>Experiments</dt>
|
|
<dd>Are all of my expected experiments from earlier addressed?</dd>
|
|
<dd>Do the experiments measure what the authors want to measure?</dd>
|
|
<dd>Are the datasets reasonable/representative of the motivating workloads?</dd>
|
|
<dd>Do the graphs support the paper's claims?</dd>
|
|
</dl>
|
|
</section>
|
|
|
|
<section>
|
|
<h3>Feedback</h3>
|
|
|
|
<p>Be Specific</p>
|
|
|
|
<ul>
|
|
<li>Communicate to the authors at least one way to address your concern.</li>
|
|
<li>Establish a clear metric that can test whether the concern is addressed.</li>
|
|
<li>Include citations where possible.</li>
|
|
<li>Differentiate suggestions from criticism.</li>
|
|
<li>Refer to specific lines of text.</li>
|
|
</ul>
|
|
</section> |