Tathagata Das
7ccbbdacb9
Made block generator thread safe to fix Kafka bug.
2013-11-12 00:10:45 -08:00
Dan Crankshaw
a13460bb64
Updated documentation
2013-11-11 23:42:02 -08:00
Dan Crankshaw
7c573a8b43
Added PartitionStrategy option
2013-11-11 23:42:01 -08:00
Dan Crankshaw
8d8056da14
Fixed issue with canonical edge partitioner.
2013-11-11 23:40:23 -08:00
Dan Crankshaw
4a670ef0ba
Merge branch 'master' of github.com:amplab/graphx
2013-11-11 21:42:08 -08:00
Dan Crankshaw
d19f2e8f3e
Removed slaves from git
2013-11-12 05:21:34 +00:00
Joey
143c01dbd6
Update README.md
...
Changing image references to master branch.
2013-11-11 19:37:16 -08:00
Ankur Dave
bc9f7eacb9
Enable stopping and starting a spot cluster
2013-11-11 17:50:31 -08:00
Reynold Xin
2e8d45032d
Merge pull request #63 from jegonzal/VertexSetCleanup
...
Cleanup of VertexSetRDD
2013-11-11 17:34:09 -08:00
Joseph E. Gonzalez
577092080c
Cleanning up documentation of VertexSetRDD.scala
2013-11-11 17:29:22 -08:00
Reynold Xin
b8e294a21b
Merge pull request #61 from ankurdave/pid2vid
...
Shuffle replicated vertex attributes efficiently in columnar format
2013-11-11 16:25:42 -08:00
Reynold Xin
3d7277ccbe
Merge pull request #55 from ankurdave/aggregateNeighbors-variants
...
Specialize mapReduceTriplets for accessing subsets of vertex attributes
2013-11-11 15:49:28 -08:00
Matei Zaharia
23b53efccd
Merge pull request #156 from haoyuan/master
...
add tachyon module
2013-11-11 12:30:02 -08:00
tgravescs
17bb9a27b2
Add mockito to the sbt build
2013-11-11 10:01:23 -06:00
Andrew xia
e13da05424
fix format error
2013-11-11 19:15:45 +08:00
Andrew xia
37d2f3749e
cut lines to less than 100
2013-11-11 15:49:32 +08:00
Andrew xia
b3208063af
Fix bug JIRA-923
2013-11-11 15:39:10 +08:00
Ankur Dave
bee1015620
Handle ClassNotFoundException from ByteCodeUtils
...
ByteCodeUtils.invokedMethod(), which we use in mapReduceTriplets, throws
a ClassNotFoundException when called with a closure defined in the
console. This commit catches the exception and conservatively assumes
the closure references all edge attributes.
2013-11-10 23:00:37 -08:00
Lian, Cheng
e2a43b3dcc
Made some changes according to suggestions from @aarondav
2013-11-11 12:21:54 +08:00
Haoyuan Li
6f455553c9
expose UI port only
2013-11-10 16:00:09 -08:00
Dan Crankshaw
60db25bded
Fixed merge conflicts.
2013-11-10 15:45:55 -08:00
Ankur Dave
d1ff1b7222
Build pid2vid structures only once, in Vid2Pid
2013-11-10 14:47:39 -08:00
Ankur Dave
502c511711
Use pid2vid for creating VTableReplicatedValues
2013-11-10 14:36:14 -08:00
Ankur Dave
53d24a973e
Fix typo
2013-11-10 14:24:38 -08:00
Ankur Dave
aa24b0bbe8
Add test for mapReduceTriplets in GraphSuite
2013-11-10 14:24:38 -08:00
Ankur Dave
bf4e45e685
Factor out VTableReplicatedValues
2013-11-10 14:24:38 -08:00
Ankur Dave
cdbd19bbee
Create all versions of vid2pid ahead of time
2013-11-10 14:10:23 -08:00
Ankur Dave
27e4355d61
Test no vertex attribute replication
2013-11-10 14:04:12 -08:00
Ankur Dave
80abc28078
Optimize mrTriplets for source-attr-only mapF using bytecode inspection
2013-11-10 14:04:12 -08:00
Joey
1a06f707e3
Merge pull request #60 from amplab/rxin
...
Looks good to me.
2013-11-10 10:54:44 -08:00
Lian, Cheng
ba55285177
Put the periodical resubmitFailedStages() call into a scheduled task
2013-11-11 01:25:35 +08:00
Matei Zaharia
58d4f6c8a5
Merge pull request #157 from rxin/kryo
...
3 Kryo related changes.
1. Call Kryo setReferences before calling user specified Kryo registrator. This is done so the user specified registrator can override the default setting.
2. Register more internal classes (MapStatus, BlockManagerId).
3. Slightly refactored the internal class registration to allocate less memory.
2013-11-10 09:23:56 -08:00
Reynold Xin
0e813cd483
Fix the hanging bug.
2013-11-09 23:29:37 -08:00
Reynold Xin
c845611fc3
Moved the Spark internal class registration for Kryo into an object, and added more classes (e.g. MapStatus, BlockManagerId) to the registration.
2013-11-09 23:00:08 -08:00
Haoyuan Li
77cedf81c7
add tachyon module
2013-11-09 22:52:23 -08:00
Reynold Xin
7c5f70d873
Call Kryo setReferences before calling user specified Kryo registrator.
2013-11-09 22:43:36 -08:00
Matei Zaharia
3efc019562
Merge pull request #147 from JoshRosen/fix-java-api-completeness-checker
...
Add spark-tools assembly to spark-class'ss classpath
This commit adds an assembly for `spark-tools` and adds it to `spark-class`'s classpath, allowing the JavaAPICompletenessChecker to be run against Spark 0.8+ with
./spark-class org.apache.spark.tools.JavaAPICompletenessChecker
Previously, this tool was run through the `run` script. I chose to add this to `run-example` because I didn't want to duplicate code in a `run-tool` script.
2013-11-09 17:53:49 -08:00
Matei Zaharia
87954d4c85
Merge pull request #154 from soulmachine/ClusterScheduler
...
Replace the thread inside ClusterScheduler.start() with an Akka scheduler
Threads are precious resources so that we shouldn't abuse them
2013-11-09 17:53:25 -08:00
Reynold Xin
f6c946206a
Merge pull request #58 from jegonzal/KryoMessages
...
Kryo messages
2013-11-09 16:14:45 -08:00
Reynold Xin
83bf1920c8
Merge pull request #155 from rxin/jobgroup
...
Don't reset job group when a new job description is set.
2013-11-09 15:40:29 -08:00
Reynold Xin
28f27097cf
Don't reset job group when a new job description is set.
2013-11-09 13:59:31 -08:00
Matei Zaharia
8af99f2356
Merge pull request #149 from tgravescs/fixSecureHdfsAccess
...
Fix secure hdfs access for spark on yarn
https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it works with secure hdfs on standalone. Fixing it at least for spark on yarn.
The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account things calling the getPartitions before sparkContext is initialized. The DAGScheduler does this as it tries to getShuffleMapStage.
2013-11-09 13:48:00 -08:00
Josh Rosen
a37ff0f1db
Add spark-tools assembly to spark-class classpath.
...
This allows the JavaAPICompletenessChecker to be
run with Spark 0.8+.
2013-11-09 13:42:45 -08:00
Matei Zaharia
72a601ec31
Merge pull request #152 from rxin/repl
...
Propagate SparkContext local properties from spark-repl caller thread to the repl execution thread.
2013-11-09 11:55:16 -08:00
soulmachine
28115fa8cb
replace the thread with a Akka scheduler
2013-11-09 22:38:27 +08:00
Lian, Cheng
765ebca04f
Remove unnecessary null checking
2013-11-09 21:13:03 +08:00
Lian, Cheng
2539c06745
Replaced the daemon thread started by DAGScheduler with an actor
2013-11-09 19:05:18 +08:00
Reynold Xin
319299941d
Propagate the SparkContext local property from the thread that calls the spark-repl to the actual execution thread.
2013-11-09 00:32:14 -08:00
Russell Cardullo
ef85a51f85
Add graphite sink for metrics
...
This adds a metrics sink for graphite. The sink must
be configured with the host and port of a graphite node
and optionally may be configured with a prefix that will
be prepended to all metrics that are sent to graphite.
2013-11-08 16:36:03 -08:00
Joseph E. Gonzalez
6083e4350f
Adding unit tests to reproduce error.
2013-11-08 15:39:30 -08:00