210 lines
17 KiB
HTML
210 lines
17 KiB
HTML
<ul>
|
||
<li><strong>Overview</strong>: Add a pre-processing phase to your system.</li>
|
||
<li><strong>Deadline</strong>: May 8</li>
|
||
<li><strong>Grade</strong>: 15% of Project Component
|
||
<ul>
|
||
<li>5% Correctness</li>
|
||
<li>5% Efficiency</li>
|
||
<li>5% Code Review</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<p style="text-align: justify;">Once again, we will be tightening performance constraints. You will be expected to complete queries in seconds, rather than tens of seconds as before. This time however, you will be given a few minutes alone with the data before we start timing you.</p>
|
||
<p style="text-align: justify;">Concretely, you will be given a period of up to 5 minutes that we'll call the Load Phase. During the load phase, you will have access to the data, as well as a database directory that will not be erased in between runs of your application. Example uses for this time include building indexes or gathering statistics about the data for use in cost-based estimation.</p>
|
||
<p style="text-align: justify;">Additionally, CREATE TABLE statements are now annotated with PRIMARY KEY and FOREIGN KEY attributes. You may hardcode index selections for the TPC-H benchmark based on your own experimentation.</p>
|
||
|
||
|
||
<hr />
|
||
|
||
<h1>BerkeleyDB</h1>
|
||
<p style="text-align: justify;">For this project, you will get access to a new library: BerkeleyDB (Java Edition). Don't let the name mislead you, BDB is not actually a full database system. Rather, BDB implements the indexing and persistence layers of a database system. Download BDB at:</p>
|
||
<p style="text-align: center;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/berkeleydb.jar">https://odin.cse.buffalo.edu/resources/berkeleydb/berkeleydb.jar</a></p>
|
||
<p style="text-align: justify;">The BerkeleyDB documentation is mirrored at:</p>
|
||
<p style="text-align: center;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/dpl.html">https://odin.cse.buffalo.edu/resources/berkeleydb/</a></p>
|
||
<p style="text-align: justify;">You can find a getting started guide at:</p>
|
||
<p style="text-align: center; font-size: 10pt;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide">https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide</a></p>
|
||
And the javadoc at:
|
||
<p style="text-align: center;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/">https://odin.cse.buffalo.edu/resources/berkeleydb/java/</a></p>
|
||
<p style="text-align: justify;">BDB can be used in two ways: The Direct Persistence layer, and the Base API. The <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/dpl.html">Direct Persistence Layer</a> is easier to use at first, as it handles index management and serialization through compiler annotations. However, this ease comes at the cost of flexibility. Especially if you plan to use secondary indexes, you may find it substantially easier to work with the <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/baseapi.html">Base API</a>. For this reason, this summary will focus on the Base API.</p>
|
||
|
||
<h1 style="text-align: justify;">Environments and Databases</h1>
|
||
<p style="text-align: justify;">A relation or table is represented in BDB as a <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/databases.html#DBOpen">Database</a>, which is grouped into units of storage called an <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/env.html">Environment</a>. The first thing that you should to do in the pre-computation phase is to <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/databases.html#DBOpen">create an Environment and one or more Databases</a>. <strong>Be absolutely sure to <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/databases.html#dbclose">close both the environment and the database</a> before you exit</strong>, as not doing so could lead to file corruption.</p>
|
||
<p style="text-align: justify;">BDB Databases are in effect clustered indexes, which means that every record stored in one is identified (and sorted by) a key. A database supports efficient access to records or ranges of records based on their keys.</p>
|
||
|
||
<h1 style="text-align: justify;">Representing, Storing, and Reading Tuples</h1>
|
||
<p style="text-align: justify;">Every tuple must be marked with a primary key, and may include one or more secondary keys. In t<span style="line-height: 1.5;">he Base API, both the value and its key are represented as a string of bytes. Both key and value must be </span><a style="line-height: 1.5;" href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/DBEntry.html#usingDbEntry">stored as a byte array encapsulated in a DatabaseEntry object</a><span style="line-height: 1.5;">. Secondary Keys are defined when creating a secondary index.</span></p>
|
||
<p style="text-align: justify;">Note that you will need to manually extract the key from the rest of the record and write some code to serialize the record and the key into byte arrays. You could use <span style="line-height: 1.5;">toString(), but you may find it substantially faster to use Java's native object serialization:</span></p>
|
||
<p style="text-align: center;"><a href="http://docs.oracle.com/javase/8/docs/api/java/io/ObjectOutputStream.html">ObjectOutputStream </a> | <a href="http://docs.oracle.com/javase/8/docs/api/java/io/ObjectInputStream.html">ObjectInputStream</a></p>
|
||
<p style="text-align: justify;">... or a pair of classes that java provides for serializing primitive data:</p>
|
||
<p style="text-align: center;"><a href="http://docs.oracle.com/javase/8/docs/api/java/io/DataOutputStream.html">DataOutputStream</a> | <a href="http://docs.oracle.com/javase/8/docs/api/java/io/DataInputStream.html">DataInputStream</a></p>
|
||
<p style="text-align: justify;">Like a Hash-Map, BDB supports a <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/usingDbt.html">simple get/put interface</a>. Tuples can be stored or looked up by their key. Like your code, BDB also provides an iterator interface called a <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/Cursors.html">Cursor</a>. Of note, BDB's cursor interface supports <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/Positioning.html#cursorsearch">index lookups</a>.</p>
|
||
|
||
<h1 style="text-align: justify;">Secondary Indexes</h1>
|
||
<p style="text-align: justify;">The Database represents a clustered index. In addition, BDB has support for unclustered indexes, which it calls <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/indexes.html">SecondaryDatabases</a>. As an unclustered index, a secondary database doesn't dictate how the tuples themselves are laid out, but still allows for (mostly) efficient lookups for secondary "keys". The term "keys" is in quotation marks, because unlike the primary key used in the primary database, a secondary database allows for multiple records with the same secondary key.</p>
|
||
<p style="text-align: justify;">To automate the management process, a secondary index is defined using an implementation of <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/keyCreator.html">SecondaryKeyCreator</a>. This class should map record DatabaseEntry objects to a (not necessarily unique) DatabaseEntry object that acts as a secondary key.</p>
|
||
|
||
<h1 style="text-align: justify;">BDB Joins</h1>
|
||
<p style="text-align: justify;">Another misnomer, BDB allows you to define so-called <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/GettingStartedGuide/joins.html">Join Cursors</a>. This is <strong>not</strong> a relational join in the traditional sense. Rather, a Join Cursor allows you to define multiple <strong>equality</strong> predicates over the base relation and scan over all records that match all of the specified lookup conditions.</p>
|
||
|
||
<h1 style="text-align: justify;">Performance Tuning</h1>
|
||
<p style="text-align: justify;">BerkeleyDB can be quite tricky to get performance out of. There are a number of options, and ways of interacting with it that can help you get the most out of this indexing software. Since evaluation on the grading boxes takes time due to the end-to-end testing process, I encourage you to evaluate on your own machines. For best results, be sure to store your database on an HDD (Results from SSDs will not be representative of the grading boxes). Recall that the grader boxes have 4 GB of RAM.</p>
|
||
|
||
<h2 style="text-align: justify;">Heap Scans</h2>
|
||
<p style="text-align: justify;">Depending on how you've implemented deserialization of the raw data files, you may find it faster to read directly from the clustered index rather than from the data file. In the reference implementation, reading from a clustered index is about twice as fast as from a data file, but this performance boost stems from several factors. If you choose to do this, take a look at <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/DiskOrderedCursor.html">DiskOrderedCursor</a>, which my experiments show is roughly about twice as fast as a regular in-order Cursor on an HDD on a fully compacted relation.</p>
|
||
|
||
<h2 style="text-align: justify;">Locking Policies</h2>
|
||
<p style="text-align: justify;">Locking is slow. Consistency is slow. As long as you're not implementing your code multithreaded or with updates or transactions, you'll find that cursor operations will be faster under <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/LockMode.html">LockMode</a>.<a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/LockMode.html#READ_UNCOMMITTED">READ_UNCOMMITTED</a>. See below for ways to set this parameter globally.</p>
|
||
|
||
<h2 style="text-align: justify;">Config Options</h2>
|
||
<p style="text-align: justify;">BDB also has numerous options that will affect the performance of your system. Several options you may wish to evaluate, both for the load and run phases:</p>
|
||
|
||
<ul>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/EnvironmentConfig.html">EnvironmentConfig</a>
|
||
<ul>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/EnvironmentMutableConfig.html#setCachePercent(int)">setCachePercent</a></li>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/EnvironmentConfig.html#setLocking(boolean)">setLocking</a></li>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/EnvironmentConfig.html#setConfigParam(java.lang.String,%20java.lang.String)">setConfigParam</a>
|
||
<ul>
|
||
<li style="text-align: justify;">EnvironmentConfig.<a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/EnvironmentConfig.html#ENV_RUN_CLEANER">ENV_RUN_CLEANER</a></li>
|
||
<li style="text-align: justify;">EnvironmentConfig.<a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/EnvironmentConfig.html#ENV_RUN_CHECKPOINTER">ENV_RUN_CHECKPOINTER</a>
|
||
<ul>
|
||
<li style="text-align: justify;">See the documentation for Environment.<a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/Environment.html#cleanLog()">cleanLog</a>() if you plan to turn either of these off.</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
</li>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/DatabaseConfig.html">DatabaseConfig</a> and <a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/SecondaryConfig.html">SecondaryConfig</a>
|
||
<ul>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/DatabaseConfig.html#setReadOnly(boolean)">setReadOnly</a></li>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/DatabaseConfig.html#setTransactional(boolean)">setTransactional</a></li>
|
||
</ul>
|
||
</li>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/DiskOrderedCursorConfig.html">DiskOrderedCursorConfig</a>
|
||
<ul>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/DiskOrderedCursorConfig.html#setInternalMemoryLimit(long)">setInternalMemoryLimit</a></li>
|
||
<li style="text-align: justify;"><a href="https://odin.cse.buffalo.edu/resources/berkeleydb/java/com/sleepycat/je/DiskOrderedCursorConfig.html#setQueueSize(int)">setQueueSize</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
|
||
<hr />
|
||
|
||
<h1>Interface</h1>
|
||
<p style="text-align: justify;">Your code will be evaluated in exactly the same way as Projects 1 and 2. Your code will be presented with a 500MB (SF 0.5) TPC-H dataset. Before grading begins, your code will be run once to preprocess the data. You will have up to 5 minutes, after which your process will be killed (if it has not yet terminated). Your code will then be run on the test suite.</p>
|
||
<p style="text-align: justify;">As before, your code will be invoked with the data directory and the relevant SQL files. Two additional parameters will be used in the preprocessing stage:</p>
|
||
|
||
<ul>
|
||
<li><tt>--db directory</tt>: A directory in which it is safe to persist data. The contents of this directory will be persisted across the entire grading run.</li>
|
||
<li><tt>--load</tt>: This parameter will be passed during the preprocessing phase. When it appears on the command line, you have up to 5 minutes to preprocess the data.</li>
|
||
</ul>
|
||
<pre>java -cp build:jsqlparser.jar:...
|
||
edu.buffalo.cse562.Main
|
||
--data [data]
|
||
--db [db]
|
||
--load
|
||
[sqlfile1] [sqlfile2] ...</pre>
|
||
This example uses the following directories and files:
|
||
<ul>
|
||
<li><tt>[data]</tt>: Table data stored in '|' separated files. As before, table names match the names provided in the matching CREATE TABLE with the .dat suffix.</li>
|
||
<li><tt>[db]</tt>: A directory for permanent data files. This directory will be persisted across all runs of the</li>
|
||
<li><tt>[sqlfileX]</tt>: A file containing CREATE TABLE and SELECT statements, defining the schema of the dataset and the query to process. If --load appears on the command line, these files will contain only CREATE TABLE statements.</li>
|
||
</ul>
|
||
<h1>Grading</h1>
|
||
<p style="text-align: justify;">Your code will be subjected to a sequence of test cases and evaluated on speed and correctness.</p>
|
||
|
||
<ul>
|
||
<li style="text-align: justify;"><strong>0/10 (F)</strong>: Your submission does not compile, does not produce correct output, or fails in some other way. Resubmission is highly encouraged.</li>
|
||
<li style="text-align: justify;"><strong>5/10 (C)</strong>: Your submission runs the test query faster than the C threshold (listed below for each query), and produces the correct output.</li>
|
||
<li style="text-align: justify;"><strong>7.5/10 (B)</strong>: Your submission runs the test query faster than the B threshold (listed below for each query), and produces the correct output.</li>
|
||
<li><strong>10/10 (A)</strong>: Your submission runs the test query faster than the A threshold (listed below for each query), and produces the correct output.</li>
|
||
</ul>
|
||
<table>
|
||
<tbody>
|
||
<tr>
|
||
<th style="text-align: center;">TPC-H Query</th>
|
||
<th style="text-align: center;">Grade</th>
|
||
<th style="text-align: center;">Maximum Run-Time (s)</th>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;" rowspan="3">Q1</th>
|
||
<th style="text-align: center;">A</th>
|
||
<td style="text-align: center;">30</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">B</th>
|
||
<td style="text-align: center;">60</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">C</th>
|
||
<td style="text-align: center;">90</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;" rowspan="3">Q3</th>
|
||
<th style="text-align: center;">A</th>
|
||
<td style="text-align: center;">5</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">B</th>
|
||
<td style="text-align: center;">30</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">C</th>
|
||
<td style="text-align: center;">120</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;" rowspan="3">Q5</th>
|
||
<th style="text-align: center;">A</th>
|
||
<td style="text-align: center;">10</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">B</th>
|
||
<td style="text-align: center;">60</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">C</th>
|
||
<td style="text-align: center;">120</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;" rowspan="3">Q6</th>
|
||
<th style="text-align: center;">A</th>
|
||
<td style="text-align: center;">20</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">B</th>
|
||
<td style="text-align: center;">45</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">C</th>
|
||
<td style="text-align: center;">70</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;" rowspan="3">Q10</th>
|
||
<th style="text-align: center;">A</th>
|
||
<td style="text-align: center;">10</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">B</th>
|
||
<td style="text-align: center;">30</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">C</th>
|
||
<td style="text-align: center;">90</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;" rowspan="3">Q12</th>
|
||
<th style="text-align: center;">A</th>
|
||
<td style="text-align: center;">40</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">B</th>
|
||
<td style="text-align: center;">60</td>
|
||
</tr>
|
||
<tr>
|
||
<th style="text-align: center;">C</th>
|
||
<td style="text-align: center;">90</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|