spark-instrumented-optimizer/streaming
Tathagata Das ae27c0ef59 [SPARK-7139] [STREAMING] Allow received block metadata to be saved to WAL and recovered on driver failure
- Enabled ReceivedBlockTracker WAL by default
- Stored block metadata in the WAL
- Optimized WALBackedBlockRDD by skipping block fetch when the block is known to not exist in Spark

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #5732 from tdas/SPARK-7139 and squashes the following commits:

575476e [Tathagata Das] Added more tests to get 100% coverage of the WALBackedBlockRDD
19668ba [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-7139
685fab3 [Tathagata Das] Addressed comments in PR
637bc9c [Tathagata Das] Changed segment to handle
466212c [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-7139
5f67a59 [Tathagata Das] Fixed HdfsUtils to handle append in local file system
1bc5bc3 [Tathagata Das] Fixed bug on unexpected recovery
d06fa21 [Tathagata Das] Enabled ReceivedBlockTracker by default, stored block metadata and optimized block fetching in WALBackedBlockRDD

(cherry picked from commit 1854ac326a)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
2015-05-05 01:46:07 -07:00
..
src [SPARK-7139] [STREAMING] Allow received block metadata to be saved to WAL and recovered on driver failure 2015-05-05 01:46:07 -07:00
pom.xml SPARK-6433 hive tests to import spark-sql test JAR for QueryTest access 2015-04-01 16:26:54 +01:00