Lian, Cheng
e2a43b3dcc
Made some changes according to suggestions from @aarondav
2013-11-11 12:21:54 +08:00
Haoyuan Li
6f455553c9
expose UI port only
2013-11-10 16:00:09 -08:00
Dan Crankshaw
60db25bded
Fixed merge conflicts.
2013-11-10 15:45:55 -08:00
Ankur Dave
d1ff1b7222
Build pid2vid structures only once, in Vid2Pid
2013-11-10 14:47:39 -08:00
Ankur Dave
502c511711
Use pid2vid for creating VTableReplicatedValues
2013-11-10 14:36:14 -08:00
Ankur Dave
53d24a973e
Fix typo
2013-11-10 14:24:38 -08:00
Ankur Dave
aa24b0bbe8
Add test for mapReduceTriplets in GraphSuite
2013-11-10 14:24:38 -08:00
Ankur Dave
bf4e45e685
Factor out VTableReplicatedValues
2013-11-10 14:24:38 -08:00
Ankur Dave
cdbd19bbee
Create all versions of vid2pid ahead of time
2013-11-10 14:10:23 -08:00
Ankur Dave
27e4355d61
Test no vertex attribute replication
2013-11-10 14:04:12 -08:00
Ankur Dave
80abc28078
Optimize mrTriplets for source-attr-only mapF using bytecode inspection
2013-11-10 14:04:12 -08:00
Joey
1a06f707e3
Merge pull request #60 from amplab/rxin
...
Looks good to me.
2013-11-10 10:54:44 -08:00
Lian, Cheng
ba55285177
Put the periodical resubmitFailedStages() call into a scheduled task
2013-11-11 01:25:35 +08:00
Matei Zaharia
58d4f6c8a5
Merge pull request #157 from rxin/kryo
...
3 Kryo related changes.
1. Call Kryo setReferences before calling user specified Kryo registrator. This is done so the user specified registrator can override the default setting.
2. Register more internal classes (MapStatus, BlockManagerId).
3. Slightly refactored the internal class registration to allocate less memory.
2013-11-10 09:23:56 -08:00
Reynold Xin
0e813cd483
Fix the hanging bug.
2013-11-09 23:29:37 -08:00
Reynold Xin
c845611fc3
Moved the Spark internal class registration for Kryo into an object, and added more classes (e.g. MapStatus, BlockManagerId) to the registration.
2013-11-09 23:00:08 -08:00
Haoyuan Li
77cedf81c7
add tachyon module
2013-11-09 22:52:23 -08:00
Reynold Xin
7c5f70d873
Call Kryo setReferences before calling user specified Kryo registrator.
2013-11-09 22:43:36 -08:00
Matei Zaharia
3efc019562
Merge pull request #147 from JoshRosen/fix-java-api-completeness-checker
...
Add spark-tools assembly to spark-class'ss classpath
This commit adds an assembly for `spark-tools` and adds it to `spark-class`'s classpath, allowing the JavaAPICompletenessChecker to be run against Spark 0.8+ with
./spark-class org.apache.spark.tools.JavaAPICompletenessChecker
Previously, this tool was run through the `run` script. I chose to add this to `run-example` because I didn't want to duplicate code in a `run-tool` script.
2013-11-09 17:53:49 -08:00
Matei Zaharia
87954d4c85
Merge pull request #154 from soulmachine/ClusterScheduler
...
Replace the thread inside ClusterScheduler.start() with an Akka scheduler
Threads are precious resources so that we shouldn't abuse them
2013-11-09 17:53:25 -08:00
Reynold Xin
f6c946206a
Merge pull request #58 from jegonzal/KryoMessages
...
Kryo messages
2013-11-09 16:14:45 -08:00
Reynold Xin
83bf1920c8
Merge pull request #155 from rxin/jobgroup
...
Don't reset job group when a new job description is set.
2013-11-09 15:40:29 -08:00
Reynold Xin
28f27097cf
Don't reset job group when a new job description is set.
2013-11-09 13:59:31 -08:00
Matei Zaharia
8af99f2356
Merge pull request #149 from tgravescs/fixSecureHdfsAccess
...
Fix secure hdfs access for spark on yarn
https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it works with secure hdfs on standalone. Fixing it at least for spark on yarn.
The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account things calling the getPartitions before sparkContext is initialized. The DAGScheduler does this as it tries to getShuffleMapStage.
2013-11-09 13:48:00 -08:00
Josh Rosen
a37ff0f1db
Add spark-tools assembly to spark-class classpath.
...
This allows the JavaAPICompletenessChecker to be
run with Spark 0.8+.
2013-11-09 13:42:45 -08:00
Matei Zaharia
72a601ec31
Merge pull request #152 from rxin/repl
...
Propagate SparkContext local properties from spark-repl caller thread to the repl execution thread.
2013-11-09 11:55:16 -08:00
soulmachine
28115fa8cb
replace the thread with a Akka scheduler
2013-11-09 22:38:27 +08:00
Lian, Cheng
765ebca04f
Remove unnecessary null checking
2013-11-09 21:13:03 +08:00
Lian, Cheng
2539c06745
Replaced the daemon thread started by DAGScheduler with an actor
2013-11-09 19:05:18 +08:00
Reynold Xin
319299941d
Propagate the SparkContext local property from the thread that calls the spark-repl to the actual execution thread.
2013-11-09 00:32:14 -08:00
Russell Cardullo
ef85a51f85
Add graphite sink for metrics
...
This adds a metrics sink for graphite. The sink must
be configured with the host and port of a graphite node
and optionally may be configured with a prefix that will
be prepended to all metrics that are sent to graphite.
2013-11-08 16:36:03 -08:00
Joseph E. Gonzalez
6083e4350f
Adding unit tests to reproduce error.
2013-11-08 15:39:30 -08:00
Aaron Davidson
dd63c548c2
Use SPARK_HOME instead of user.dir in ExecutorRunnerTest
2013-11-08 12:51:05 -08:00
tgravescs
13a19505e4
Don't call the doAs if user is unknown or the same user that is already running
2013-11-08 12:04:09 -06:00
tgravescs
f95cb04e40
Remove the runAsUser as it breaks secure hdfs access
2013-11-08 10:07:15 -06:00
tgravescs
5f9ed51719
Fix access to Secure HDFS
2013-11-08 08:41:57 -06:00
Joseph E. Gonzalez
161784d0e6
Fixing tests
2013-11-07 20:40:21 -08:00
Joseph E. Gonzalez
e523f0d2fb
merged and debugged
2013-11-07 20:19:49 -08:00
Joseph E. Gonzalez
908e606473
Additional optimizations
2013-11-07 19:47:30 -08:00
Reynold Xin
bac7be30cd
Made more specialized messages.
2013-11-07 19:39:48 -08:00
Reynold Xin
64ad3b18d9
Merge branch 'master' into rxin
...
Conflicts:
graph/src/main/scala/org/apache/spark/graph/impl/GraphImpl.scala
2013-11-07 19:23:42 -08:00
Reynold Xin
2406bf33e4
Use custom serializer for aggregation messages when the data type is int/double.
2013-11-07 19:18:58 -08:00
Ankur Dave
6ee05be1c8
Merge pull request #49 from jegonzal/graphxshell
...
GraphX Console with Logo Text
2013-11-07 19:12:41 -08:00
Ankur Dave
a9f96b54e4
Merge pull request #56 from jegonzal/PregelAPIChanges
...
Changing Pregel API to use mapReduceTriplets instead of aggregateNeighbors
2013-11-07 18:56:56 -08:00
Joseph E. Gonzalez
e9308e0e75
Changing Pregel API to operate directly on edge triplets in SendMessage rather than (Vid, EdgeTriplet) pairs.
2013-11-07 18:04:06 -08:00
Reynold Xin
5907137d11
Merge pull request #54 from amplab/rxin
...
Converted for loops to while loops in EdgePartition.
2013-11-07 16:58:31 -08:00
Reynold Xin
6fadff2b92
Converted for loops to while loops in EdgePartition.
2013-11-07 16:54:33 -08:00
Reynold Xin
edf41647f4
Merge pull request #53 from amplab/rxin
...
Added GraphX to classpath.
2013-11-07 16:22:43 -08:00
Reynold Xin
95f1f5315e
Added GraphX to classpath.
2013-11-07 16:22:05 -08:00
Reynold Xin
c379e10455
Merge pull request #51 from jegonzal/VertexSetRDD
...
Reverting to Array based (materialized) output in VertexSetRDD
2013-11-07 16:01:47 -08:00
