ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4975 commits 6 branches 181 tags 397 MiB
Commit graph

272 commits

Author SHA1 Message Date
Ankur Dave 5e20cbaf66 Define localVidMap once per VTableReplicated 2013-12-14 15:28:22 -08:00
Ankur Dave dc72147900 Fix bug in interaction of incr. view maint., skipStale, and join rewrite 2013-12-14 15:28:22 -08:00
Ankur Dave d076fe5c9e Finish up 49f7fc5ed2edd42b118182438d84ed7d4e7d521f 2013-12-14 15:28:22 -08:00
Ankur Dave 4f4f0add24 Fix bug in mapVertices and outerJoinVertices 2013-12-14 15:28:22 -08:00
Ankur Dave 6bf51a5e15 Don't expose {src,dst}Mask in EdgeTriplet 
Exposing this information is not necessary due to the addition of
skipStale.
 2013-12-14 15:28:22 -08:00
Ankur Dave 87f2909561 Incremental view maintenance for all graph ops 
All GraphImpl operations now support incremental maintenance of the
replicated vertex view by reusing vTableReplicated whenever
possible. This has two consequences:

- This is not confined to the deltaJoinVertices operator anymore,
  so we rename it to the more intuitive "updateVertices" instead.

- This enables mrTriplets to support skipStale.
 2013-12-14 15:28:22 -08:00
Ankur Dave c3e0f01ad9 Remove calls to ClosureCleaner in GraphX 
All Spark RDD methods already clean closures so we don't need to.
 2013-12-14 15:28:21 -08:00
Ankur Dave 5b0d6f0ad5 Remove static Pregel; take maxIters in dynamic Pregel 2013-12-12 18:03:19 -08:00
Dan Crankshaw 12483d4ae6 Added BoundedPriorityQueue kryo registrator. Fixes top issue. 2013-12-11 20:56:21 -08:00
Ankur Dave 84d0e1a334 Set RDD names for easy debugging 2013-12-07 04:05:45 -08:00
Reynold Xin 41721b1494 Fixed a bug in VTableReplicated that we only process the first block. 2013-12-06 00:51:12 -08:00
Reynold Xin 3b0ee53eda Minor update. 2013-12-05 23:30:56 -08:00
Reynold Xin 15168d6c4d Fixed a bug in VTableReplicated that we are always broadcasting all the vertices. 2013-12-05 23:25:53 -08:00
Reynold Xin a6075ba11f Merge branch 'pregel-replicate-changed' of github.com:ankurdave/graphx into pregel-replicate-changed 2013-12-05 22:35:16 -08:00
Ankur Dave b707861ba0 Simplify GraphImpl.deltaJoinVertices 2013-12-05 20:01:32 -08:00
Ankur Dave 67bfa7fd65 Test prevVTableReplicated code path 2013-12-05 19:16:00 -08:00
Reynold Xin 920282c36a Merge branch 'pregel-replicate-changed' of github.com:ankurdave/graphx into pregel-replicate-changed 2013-12-05 18:57:36 -08:00
Ankur Dave 39b0256668 Use mask for dynamic Pregel 2013-12-05 18:55:59 -08:00
Ankur Dave 7457abe282 Re-roll last iteration of static Pregel loop 2013-12-05 16:59:49 -08:00
Reynold Xin 4f80dd22bd Fixed a bug that variable encoding doesn't work for ints that use all 64 bits. 2013-12-05 16:19:37 -08:00
Ankur Dave 4d4c1c931e Test Graph.deltaJoinVertices and VertexRDD.diff 2013-12-05 16:15:56 -08:00
Ankur Dave 15374fed97 In static Pregel, replicate only changed vertices 2013-12-05 15:55:33 -08:00
Reynold Xin 3fc4534d19 wip delta join. 2013-12-05 14:55:26 -08:00
Ankur Dave 6a7b396e5d Finish work on #85 2013-12-05 12:35:03 -08:00
Ankur Dave a3bb98b88a Always write Vids using variable encoding 
Also, autoformat Serializers.scala.
 2013-12-05 12:06:07 -08:00
Ankur Dave 2e583d2de4 Declare Vids explicitly to avoid ClassCastException 2013-12-04 17:34:14 -08:00
Ankur Dave 92e96f727e Fix compile errors in GraphSuite and SerializerSuite 2013-12-04 17:29:52 -08:00
Reynold Xin 8701cb55e6 Use specialized shuffler for aggregation. 2013-12-01 21:55:50 -08:00
Reynold Xin 55edbb4209 Created an algorithms package and put all algorithms there. 2013-12-01 20:17:26 -08:00
Reynold Xin 583a389e3f Removed PartitionStrategy from GraphImpl. 2013-11-30 17:00:54 -08:00
Reynold Xin 6eeadb667d Created EdgeRDD. 2013-11-30 16:53:54 -08:00
Reynold Xin 34ee81415e Merged Ankur's pull request #80 and fixed subgraph. 2013-11-30 15:10:30 -08:00
Reynold Xin 8e790b7f7a Merge branch 'subgraph-test' of github.com:ankurdave/graphx into clean1 
Conflicts:
	graph/src/main/scala/org/apache/spark/graph/impl/VertexPartition.scala
 2013-11-30 14:48:43 -08:00
Reynold Xin 229022891f Made all VertexPartition internal data structures private. 2013-11-30 14:45:56 -08:00
Reynold Xin b30e0ae035 Added an optimized count to VertexSetRDD. 2013-11-30 14:24:18 -08:00
Reynold Xin 689f757f7a Merge branch 'clean1' of github.com:amplab/graphx into clean1 2013-11-30 14:16:06 -08:00
Reynold Xin 4d3d68b8fb Minor update to tests. 2013-11-30 14:15:47 -08:00
Ankur Dave 3292cb0f9c Revert "Fix join error by caching vTable in mapReduceTriplets" 
This reverts commit dee1318d3d, which is
unnecessary due to 7528e6d5f1.
 2013-11-30 14:05:32 -08:00
Reynold Xin e72bd91590 Merge branch 'clean1' of github.com:amplab/graphx into clean1 2013-11-30 14:04:45 -08:00
Reynold Xin 7528e6d5f1 Enable joining arbitrary VertexPartitions (with different indexes). 2013-11-30 14:04:16 -08:00
Ankur Dave eed3195038 Fix VertexSetRDD test by enabling index reuse 2013-11-30 13:50:37 -08:00
Ankur Dave dee1318d3d Fix join error by caching vTable in mapReduceTriplets 2013-11-30 13:37:19 -08:00
Reynold Xin 10c0f9b0bb Added a log4j properties file for graphx unit tests. 2013-11-30 13:18:43 -08:00
Reynold Xin 95e83af209 More, bigger cleaning for better encapsulation of VertexSetRDD and VertexPartition. This is work in progress as stuff doesn't really run. 2013-11-27 00:30:26 -08:00
Reynold Xin caba162861 Added join and aggregateUsingIndex to VertexPartition. 2013-11-26 21:02:39 -08:00
Ankur Dave 9e896be375 Test edge filtering in subgraph (test fails) 2013-11-26 15:58:55 -08:00
Ankur Dave 137294e2ab Test GraphImpl.subgraph and fix bug 2013-11-26 15:32:47 -08:00
Reynold Xin 2d19d0381b Merge branch 'simplify' into clean 2013-11-26 13:55:26 -08:00
Reynold Xin d58bfa8573 Code cleaning to improve readability. 2013-11-26 13:54:46 -08:00
Dan Crankshaw 4b6b15dadd Actually use partitioner command line args in Analytics. 2013-11-24 16:38:38 -08:00