Ankur Dave
b2f595f552
Test more graph ops and skipStale
2013-12-14 15:28:22 -08:00
Ankur Dave
45ea674e21
Handle diffing VertexPartitions with different indexes
2013-12-14 15:28:22 -08:00
Ankur Dave
8f4b8e9b95
Reuse previous localVidMap if available
2013-12-14 15:28:22 -08:00
Ankur Dave
5e20cbaf66
Define localVidMap once per VTableReplicated
2013-12-14 15:28:22 -08:00
Ankur Dave
dc72147900
Fix bug in interaction of incr. view maint., skipStale, and join rewrite
2013-12-14 15:28:22 -08:00
Ankur Dave
d076fe5c9e
Finish up 49f7fc5ed2edd42b118182438d84ed7d4e7d521f
2013-12-14 15:28:22 -08:00
Ankur Dave
4f4f0add24
Fix bug in mapVertices and outerJoinVertices
2013-12-14 15:28:22 -08:00
Ankur Dave
6bf51a5e15
Don't expose {src,dst}Mask in EdgeTriplet
...
Exposing this information is not necessary due to the addition of
skipStale.
2013-12-14 15:28:22 -08:00
Ankur Dave
87f2909561
Incremental view maintenance for all graph ops
...
All GraphImpl operations now support incremental maintenance of the
replicated vertex view by reusing vTableReplicated whenever
possible. This has two consequences:
- This is not confined to the deltaJoinVertices operator anymore,
so we rename it to the more intuitive "updateVertices" instead.
- This enables mrTriplets to support skipStale.
2013-12-14 15:28:22 -08:00
Ankur Dave
c3e0f01ad9
Remove calls to ClosureCleaner in GraphX
...
All Spark RDD methods already clean closures so we don't need to.
2013-12-14 15:28:21 -08:00
Reynold Xin
840af5e8e4
Merge pull request #99 from ankurdave/only-dynamic-pregel
...
Remove static Pregel; take maxIterations in dynamic Pregel
2013-12-14 12:51:51 -08:00
Ankur Dave
5b0d6f0ad5
Remove static Pregel; take maxIters in dynamic Pregel
2013-12-12 18:03:19 -08:00
Reynold Xin
ce6ca4ea61
Merge pull request #97 from dcrankshaw/fix-rddtop
...
Added BoundedPriorityQueue kryo registrator. Fixes top issue.
2013-12-11 22:30:54 -08:00
Dan Crankshaw
12483d4ae6
Added BoundedPriorityQueue kryo registrator. Fixes top issue.
2013-12-11 20:56:21 -08:00
Reynold Xin
3abfbfb104
Merge pull request #92 from ankurdave/rdd-names
...
Set RDD names for easy debugging
2013-12-07 11:24:19 -08:00
Ankur Dave
84d0e1a334
Set RDD names for easy debugging
2013-12-07 04:05:45 -08:00
Ankur Dave
31e8a14e17
Merge pull request #90 from amplab/pregel-replicate-changed
...
Replicate only changed vertices
2013-12-06 21:49:55 -08:00
Reynold Xin
41721b1494
Fixed a bug in VTableReplicated that we only process the first block.
2013-12-06 00:51:12 -08:00
Reynold Xin
3b0ee53eda
Minor update.
2013-12-05 23:30:56 -08:00
Reynold Xin
15168d6c4d
Fixed a bug in VTableReplicated that we are always broadcasting all the vertices.
2013-12-05 23:25:53 -08:00
Reynold Xin
a6075ba11f
Merge branch 'pregel-replicate-changed' of github.com:ankurdave/graphx into pregel-replicate-changed
2013-12-05 22:35:16 -08:00
Ankur Dave
b707861ba0
Simplify GraphImpl.deltaJoinVertices
2013-12-05 20:01:32 -08:00
Ankur Dave
67bfa7fd65
Test prevVTableReplicated code path
2013-12-05 19:16:00 -08:00
Reynold Xin
920282c36a
Merge branch 'pregel-replicate-changed' of github.com:ankurdave/graphx into pregel-replicate-changed
2013-12-05 18:57:36 -08:00
Ankur Dave
39b0256668
Use mask for dynamic Pregel
2013-12-05 18:55:59 -08:00
Ankur Dave
7457abe282
Re-roll last iteration of static Pregel loop
2013-12-05 16:59:49 -08:00
Ankur Dave
1c8500efc0
Merge pull request #88 from amplab/varenc
...
Fixed a bug that variable encoding doesn't work for ints that use all 64 bits.
2013-12-05 16:25:44 -08:00
Reynold Xin
4f80dd22bd
Fixed a bug that variable encoding doesn't work for ints that use all 64 bits.
2013-12-05 16:19:37 -08:00
Ankur Dave
3d57be6151
Revert changes to project/SparkBuild.scala
...
Changes were made in 3fc4534d19
2013-12-05 16:19:26 -08:00
Ankur Dave
4d4c1c931e
Test Graph.deltaJoinVertices and VertexRDD.diff
2013-12-05 16:15:56 -08:00
Ankur Dave
15374fed97
In static Pregel, replicate only changed vertices
2013-12-05 15:55:33 -08:00
Reynold Xin
3fc4534d19
wip delta join.
2013-12-05 14:55:26 -08:00
Ankur Dave
e0bcaa0942
Merge pull request #86 from ankurdave/vid-varenc
...
Finish work on #85
2013-12-05 12:37:02 -08:00
Ankur Dave
6a7b396e5d
Finish work on #85
2013-12-05 12:35:03 -08:00
Reynold Xin
3e96b9ab1e
Merge pull request #85 from ankurdave/vid-varenc
...
Always write Vids using variable encoding
2013-12-05 12:07:36 -08:00
Ankur Dave
a3bb98b88a
Always write Vids using variable encoding
...
Also, autoformat Serializers.scala.
2013-12-05 12:06:07 -08:00
Joey
e0347ba6c7
Merge pull request #83 from ankurdave/fix-tests
...
Fix compile errors in GraphSuite and SerializerSuite
2013-12-04 17:38:06 -08:00
Ankur Dave
2e583d2de4
Declare Vids explicitly to avoid ClassCastException
2013-12-04 17:34:14 -08:00
Ankur Dave
92e96f727e
Fix compile errors in GraphSuite and SerializerSuite
2013-12-04 17:29:52 -08:00
Reynold Xin
cbd3b7571f
Merge pull request #81 from amplab/clean1
...
Codebase refactoring
2013-12-04 15:35:26 -08:00
Reynold Xin
8701cb55e6
Use specialized shuffler for aggregation.
2013-12-01 21:55:50 -08:00
Reynold Xin
55edbb4209
Created an algorithms package and put all algorithms there.
2013-12-01 20:17:26 -08:00
Reynold Xin
e36fe55a03
Memoize preferred locations in ZippedPartitionsBaseRDD so preferred location computation doesn't lead to exponential explosion.
2013-11-30 18:07:36 -08:00
Reynold Xin
583a389e3f
Removed PartitionStrategy from GraphImpl.
2013-11-30 17:00:54 -08:00
Reynold Xin
6eeadb667d
Created EdgeRDD.
2013-11-30 16:53:54 -08:00
Reynold Xin
34ee81415e
Merged Ankur's pull request #80 and fixed subgraph.
2013-11-30 15:10:30 -08:00
Reynold Xin
8e790b7f7a
Merge branch 'subgraph-test' of github.com:ankurdave/graphx into clean1
...
Conflicts:
graph/src/main/scala/org/apache/spark/graph/impl/VertexPartition.scala
2013-11-30 14:48:43 -08:00
Reynold Xin
229022891f
Made all VertexPartition internal data structures private.
2013-11-30 14:45:56 -08:00
Reynold Xin
b30e0ae035
Added an optimized count to VertexSetRDD.
2013-11-30 14:24:18 -08:00
Reynold Xin
689f757f7a
Merge branch 'clean1' of github.com:amplab/graphx into clean1
2013-11-30 14:16:06 -08:00