ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4477 commits 6 branches 181 tags 397 MiB
Commit graph

4477 commits

This branch
This branch
All branches
Author SHA1 Message Date
Joseph E. Gonzalez ede329336d Fixing a scaladoc bug in graph generators. 2013-10-29 14:50:12 -07:00
Joseph E. Gonzalez 15958ca65a Reindenting documentation. 2013-10-29 14:01:24 -07:00
Joseph E. Gonzalez d316cad9b1 Documented Graph.appy functions. 2013-10-29 13:58:04 -07:00
Joseph E. Gonzalez 19da8820fc Minor modifications to documentation. 2013-10-29 11:06:06 -07:00
Joseph E. Gonzalez 77626d1507 Adding collect neighbors and documenting GraphOps. 2013-10-29 11:05:42 -07:00
Joseph E. Gonzalez d6a902f309 Finished updating connected components to used Pregel like abstraction and created a series of tests in the AnalyticsSuite. 2013-10-28 11:52:26 -07:00
Joseph E. Gonzalez a2287ae138 Implementing connected components on top of pregel like abstraction. 2013-10-27 10:42:11 -07:00
Joseph E. Gonzalez 6a0fbc0374 Updating the GraphLab API to match the changes made to the Pregel API. 2013-10-26 15:44:19 -07:00
Joseph E. Gonzalez 08024c938c Adding more documentation to the Pregel API as well as additional functionality including the ability to specify the edge direction along which messages are computed. 2013-10-26 15:42:51 -07:00
Joseph E. Gonzalez 00e73833cc Fixing a bug in reverse edge direction. 2013-10-26 15:10:30 -07:00
Reynold Xin 6f82c42690 Merge pull request #34 from jegonzal/AnalyticsCleanup 
Analytics Cleanup
 2013-10-24 11:09:46 -07:00
Joseph E. Gonzalez 9d1e5946fe Merge branch 'master' of https://github.com/amplab/graphx into AnalyticsCleanup 2013-10-23 00:26:06 -07:00
Joseph E. Gonzalez c30624dcbb Adding dynamic pregel, fixing bugs in PageRank, and adding basic analytics unit tests. 2013-10-23 00:25:45 -07:00
Joseph E. Gonzalez 0bd92ed8d0 Fixing a bug in pregel where the initial vertex-program results are lost. 2013-10-22 19:10:51 -07:00
Dan Crankshaw 49d5cdac33 Merge pull request #30 from jegonzal/VertexSetRDD_Tests 
Testing and Documenting VertexSetRDD
 2013-10-22 15:38:02 -07:00
Joseph E. Gonzalez be8269af07 Merge branch 'VertexSetRDD_Tests' into AnalyticsCleanup 2013-10-22 15:03:49 -07:00
Joseph E. Gonzalez e3eb03d5b5 Starting analytics test suite. 2013-10-22 15:03:16 -07:00
Joseph E. Gonzalez ba5c75692a Updating analytics to reflect changes in the pregel interface and moving degree information into the edge attribute. 2013-10-22 15:03:00 -07:00
Joseph E. Gonzalez 46b195253e Adding some additional graph generators to support unit testing of the analytics package. 2013-10-22 15:01:49 -07:00
Joseph E. Gonzalez 14a3329a11 Changing the Pregel interface slightly to better support type inference. 2013-10-22 15:01:20 -07:00
Joseph E. Gonzalez ebdbedc3e9 Documenting VertexSetRDD and added some testing code for VertexSetRDD 2013-10-19 01:26:08 -07:00
Joseph E. Gonzalez dbc8c9868a Fixing bug in VertexSetRDD that breaks Graph tests. 2013-10-18 23:44:06 -07:00
Reynold Xin 9cf43cfeb7 Merge pull request #28 from jegonzal/VertexSetRDD 
Refactoring IndexedRDD to VertexSetRDD.
 2013-10-18 22:07:21 -07:00
Reynold Xin f888a5b051 Merge pull request #29 from ankurdave/unit-tests 
Unit tests for Graph and GraphOps
 2013-10-18 22:06:58 -07:00
Ankur Dave 2d3603930e Add a unit test for GraphOps.joinVertices 2013-10-18 19:46:13 -07:00
Ankur Dave d15db10831 Add a unit test for Graph.mapEdges 2013-10-18 19:46:13 -07:00
Ankur Dave d429f015c0 Update GraphSuite aggregateNeighbors test 2013-10-18 19:46:13 -07:00
Joseph E. Gonzalez 5d01ebca3c Specializing IndexedRDD as VertexSetRDD. 
1) This allows the index map to be optimized for Vids
2) This makes the code more readable
2) The Graph API can now return VertexSetRDDs from operations that produce results for vertices
 2013-10-18 19:03:59 -07:00
Ankur Dave 0794bd7bc5 Merge pull request #27 from jegonzal/removed_indexedrdd_from_core 
Removing IndexedRDD changes for spark/core
 2013-10-18 18:59:58 -07:00
Joseph E. Gonzalez bb58aa5330 Added some stub code to address the case where a vertex could occur multiple times in the vertex table or where a vertex in the edge list may not appear in the vertex table. 
Moving IndexedRDD into the graphx source tree and removing dependencies in /core.
 2013-10-18 18:15:32 -07:00
Joseph E. Gonzalez fc5af50a2f Merge branch 'master' of https://github.com/amplab/graphx 2013-10-18 18:15:17 -07:00
Ankur Dave 36a902e52d Revert accidental removal of code in 3a40a5e 2013-10-18 16:19:40 -07:00
Ankur Dave 971f824014 Revert unnecessary changes to core 
While benchmarking, we accidentally committed some unnecessary changes
to core such as adding logging. These changes make it more difficult to
merge from Spark upstream, so this commit reverts them.
 2013-10-18 16:07:38 -07:00
Dan Crankshaw 8bd5f89662 Merge branch 'indexedrdd_graphx' of github.com:amplab/graphx into indexedrdd_graphx 2013-10-18 15:11:28 -07:00
Dan Crankshaw 3a40a5eb30 Added some documentation. 2013-10-18 15:11:21 -07:00
Joseph E. Gonzalez 1856b37e9d Merge branch 'master' of https://github.com/apache/incubator-spark into indexedrdd_graphx 2013-10-18 12:21:19 -07:00
Joseph E. Gonzalez e028079b0f Merging with spark upstream changes. 2013-10-18 12:02:14 -07:00
Joseph E. Gonzalez 3f3d28c73f Switching from Seq to IndexedSeq 2013-10-17 19:55:36 -07:00
Joseph E. Gonzalez 9a03c5fe28 This commit accomplishes three goals: 
1) Further simplification of the IndexedRDD operations (eliminating some)
 2) Aggressive reuse of HashMaps
 3) Pipelining join operations within indexedrdd
 2013-10-17 19:01:48 -07:00
Joey 099977fd1b Merge pull request #26 from ankurdave/split-vTableReplicated 
Great work!
 2013-10-17 14:17:08 -07:00
Matei Zaharia fc26e5b832 Merge pull request #69 from KarthikTunga/master 
Fix for issue SPARK-627. Implementing --config argument in the scripts.

This code fix is for issue SPARK-627. I added code to consider --config arguments in the scripts. In case the  <conf-dir> is not a directory the scripts exit. I removed the --hosts argument. It can be achieved by giving a different config directory. Let me know if an explicit --hosts argument is required.
 2013-10-17 13:21:07 -07:00
Ankur Dave bf19aac2b7 Use ArrayBuilder instead of ArrayBuffer 
ArrayBuilder is specialized for holding primitive VD types.
 2013-10-17 13:19:00 -07:00
Matei Zaharia cf64f63f8a Merge pull request #67 from kayousterhout/remove_tsl 
Removed TaskSchedulerListener interface.

The interface was used only by the DAG scheduler (so it wasn't necessary
to define the additional interface), and the naming makes it very
confusing when reading the code (because "listener" was used
to describe the DAG scheduler, rather than SparkListeners, which
implement a nearly-identical interface but serve a different
function).

@mateiz - is there a reason for this interface that I'm missing?
 2013-10-17 11:12:28 -07:00
Ankur Dave 2282d27cf1 Cache msgsByPartition 2013-10-16 23:56:15 -07:00
Kay Ousterhout 809f547633 Fixed unit tests 2013-10-16 23:16:12 -07:00
KarthikTunga 8537f19268 SPARK-627 , Implementing --config arguments in the scripts 2013-10-16 23:00:33 -07:00
KarthikTunga ff4fb1f7ee SPARK-627 , Implementing --config arguments in the scripts 2013-10-16 22:55:15 -07:00
KarthikTunga a32aa6b351 Implementing --config argument in the scripts 2013-10-16 22:51:09 -07:00
Ankur Dave bc234bf0e1 Split vTableReplicated into two RDDs 
Previously, (vTableReplicated: IndexedRDD[Pid, VertexHashMap[VD]])
stored one hashmap per partition, taking Vid directly to VD.

To take advantage of rxin's new hashmaps (see
rxin/incubator-spark@32a79d6d13), this
commit splits that data structure into two RDDs:

(vTableReplicationMap: IndexedRDD[Pid, VertexIdToIndexMap]) stores a map
per partition from vertex ID to the index where that vertex's attribute
is stored. This index refers to an array in the same partition in
vTableReplicatedValues.

(vTableReplicatedValues: IndexedRDD[Pid, Array[VD]]) stores the vertex
data and is arranged as described above.
 2013-10-16 19:22:23 -07:00
Ankur Dave af8e461841 Set serialization properties in GraphSuite 2013-10-16 19:21:24 -07:00