ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4960 commits 6 branches 181 tags 397 MiB
Commit graph

4960 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia dfd1ebc2d1 Merge pull request #164 from tdas/kafka-fix 
Made block generator thread safe to fix Kafka bug.

This is a very important bug fix. Data can and was being lost in the kafka due to this.
 2013-11-12 09:10:05 -08:00
Tathagata Das 7ccbbdacb9 Made block generator thread safe to fix Kafka bug. 2013-11-12 00:10:45 -08:00
Dan Crankshaw a13460bb64 Updated documentation 2013-11-11 23:42:02 -08:00
Dan Crankshaw 7c573a8b43 Added PartitionStrategy option 2013-11-11 23:42:01 -08:00
Dan Crankshaw 8d8056da14 Fixed issue with canonical edge partitioner. 2013-11-11 23:40:23 -08:00
Dan Crankshaw 4a670ef0ba Merge branch 'master' of github.com:amplab/graphx 2013-11-11 21:42:08 -08:00
Dan Crankshaw d19f2e8f3e Removed slaves from git 2013-11-12 05:21:34 +00:00
Joey 143c01dbd6 Update README.md 
Changing image references to master branch.
 2013-11-11 19:37:16 -08:00
Ankur Dave bc9f7eacb9 Enable stopping and starting a spot cluster 2013-11-11 17:50:31 -08:00
Reynold Xin 2e8d45032d Merge pull request #63 from jegonzal/VertexSetCleanup 
Cleanup of VertexSetRDD
 2013-11-11 17:34:09 -08:00
Joseph E. Gonzalez 577092080c Cleanning up documentation of VertexSetRDD.scala 2013-11-11 17:29:22 -08:00
Reynold Xin b8e294a21b Merge pull request #61 from ankurdave/pid2vid 
Shuffle replicated vertex attributes efficiently in columnar format
 2013-11-11 16:25:42 -08:00
Reynold Xin 3d7277ccbe Merge pull request #55 from ankurdave/aggregateNeighbors-variants 
Specialize mapReduceTriplets for accessing subsets of vertex attributes
 2013-11-11 15:49:28 -08:00
Matei Zaharia 23b53efccd Merge pull request #156 from haoyuan/master 
add tachyon module
 2013-11-11 12:30:02 -08:00
tgravescs 17bb9a27b2 Add mockito to the sbt build 2013-11-11 10:01:23 -06:00
Andrew xia e13da05424 fix format error 2013-11-11 19:15:45 +08:00
Andrew xia 37d2f3749e cut lines to less than 100 2013-11-11 15:49:32 +08:00
Andrew xia b3208063af Fix bug JIRA-923 2013-11-11 15:39:10 +08:00
Ankur Dave bee1015620 Handle ClassNotFoundException from ByteCodeUtils 
ByteCodeUtils.invokedMethod(), which we use in mapReduceTriplets, throws
a ClassNotFoundException when called with a closure defined in the
console. This commit catches the exception and conservatively assumes
the closure references all edge attributes.
 2013-11-10 23:00:37 -08:00
Lian, Cheng e2a43b3dcc Made some changes according to suggestions from @aarondav 2013-11-11 12:21:54 +08:00
Haoyuan Li 6f455553c9 expose UI port only 2013-11-10 16:00:09 -08:00
Dan Crankshaw 60db25bded Fixed merge conflicts. 2013-11-10 15:45:55 -08:00
Ankur Dave d1ff1b7222 Build pid2vid structures only once, in Vid2Pid 2013-11-10 14:47:39 -08:00
Ankur Dave 502c511711 Use pid2vid for creating VTableReplicatedValues 2013-11-10 14:36:14 -08:00
Ankur Dave 53d24a973e Fix typo 2013-11-10 14:24:38 -08:00
Ankur Dave aa24b0bbe8 Add test for mapReduceTriplets in GraphSuite 2013-11-10 14:24:38 -08:00
Ankur Dave bf4e45e685 Factor out VTableReplicatedValues 2013-11-10 14:24:38 -08:00
Ankur Dave cdbd19bbee Create all versions of vid2pid ahead of time 2013-11-10 14:10:23 -08:00
Ankur Dave 27e4355d61 Test no vertex attribute replication 2013-11-10 14:04:12 -08:00
Ankur Dave 80abc28078 Optimize mrTriplets for source-attr-only mapF using bytecode inspection 2013-11-10 14:04:12 -08:00
Joey 1a06f707e3 Merge pull request #60 from amplab/rxin 
Looks good to me.
 2013-11-10 10:54:44 -08:00
Lian, Cheng ba55285177 Put the periodical resubmitFailedStages() call into a scheduled task 2013-11-11 01:25:35 +08:00
Matei Zaharia 58d4f6c8a5 Merge pull request #157 from rxin/kryo 
3 Kryo related changes.

1. Call Kryo setReferences before calling user specified Kryo registrator. This is done so the user specified registrator can override the default setting.

2. Register more internal classes (MapStatus, BlockManagerId).

3. Slightly refactored the internal class registration to allocate less memory.
 2013-11-10 09:23:56 -08:00
Reynold Xin 0e813cd483 Fix the hanging bug. 2013-11-09 23:29:37 -08:00
Reynold Xin c845611fc3 Moved the Spark internal class registration for Kryo into an object, and added more classes (e.g. MapStatus, BlockManagerId) to the registration. 2013-11-09 23:00:08 -08:00
Haoyuan Li 77cedf81c7 add tachyon module 2013-11-09 22:52:23 -08:00
Reynold Xin 7c5f70d873 Call Kryo setReferences before calling user specified Kryo registrator. 2013-11-09 22:43:36 -08:00
Matei Zaharia 3efc019562 Merge pull request #147 from JoshRosen/fix-java-api-completeness-checker 
Add spark-tools assembly to spark-class'ss classpath

This commit adds an assembly for `spark-tools` and adds it to `spark-class`'s classpath, allowing the JavaAPICompletenessChecker to be run against Spark 0.8+ with

    ./spark-class org.apache.spark.tools.JavaAPICompletenessChecker

Previously, this tool was run through the `run` script.  I chose to add this to `run-example` because I didn't want to duplicate code in a `run-tool` script.
 2013-11-09 17:53:49 -08:00
Matei Zaharia 87954d4c85 Merge pull request #154 from soulmachine/ClusterScheduler 
Replace the thread inside ClusterScheduler.start() with an Akka scheduler

Threads are precious resources so that we shouldn't abuse them
 2013-11-09 17:53:25 -08:00
Reynold Xin f6c946206a Merge pull request #58 from jegonzal/KryoMessages 
Kryo messages
 2013-11-09 16:14:45 -08:00
Reynold Xin 83bf1920c8 Merge pull request #155 from rxin/jobgroup 
Don't reset job group when a new job description is set.
 2013-11-09 15:40:29 -08:00
Reynold Xin 28f27097cf Don't reset job group when a new job description is set. 2013-11-09 13:59:31 -08:00
Matei Zaharia 8af99f2356 Merge pull request #149 from tgravescs/fixSecureHdfsAccess 
Fix secure hdfs access for spark on yarn

https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it works with secure hdfs on standalone. Fixing it at least for spark on yarn.

The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account things calling the getPartitions before sparkContext is initialized. The DAGScheduler does this as it tries to getShuffleMapStage.
 2013-11-09 13:48:00 -08:00
Josh Rosen a37ff0f1db Add spark-tools assembly to spark-class classpath. 
This allows the JavaAPICompletenessChecker to be
run with Spark 0.8+.
 2013-11-09 13:42:45 -08:00
Matei Zaharia 72a601ec31 Merge pull request #152 from rxin/repl 
Propagate SparkContext local properties from spark-repl caller thread to the repl execution thread.
 2013-11-09 11:55:16 -08:00
soulmachine 28115fa8cb replace the thread with a Akka scheduler 2013-11-09 22:38:27 +08:00
Lian, Cheng 765ebca04f Remove unnecessary null checking 2013-11-09 21:13:03 +08:00
Lian, Cheng 2539c06745 Replaced the daemon thread started by DAGScheduler with an actor 2013-11-09 19:05:18 +08:00
Reynold Xin 319299941d Propagate the SparkContext local property from the thread that calls the spark-repl to the actual execution thread. 2013-11-09 00:32:14 -08:00
Russell Cardullo ef85a51f85 Add graphite sink for metrics 
This adds a metrics sink for graphite.  The sink must
be configured with the host and port of a graphite node
and optionally may be configured with a prefix that will
be prepended to all metrics that are sent to graphite.
 2013-11-08 16:36:03 -08:00