ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5060 commits 6 branches 181 tags 397 MiB
Commit graph

5060 commits

This branch
This branch
All branches
Author SHA1 Message Date
Reynold Xin ded10ce5b0 Merge pull request #103 from amplab/optimizations 
Optimizations cherry-picked from SIGMOD branches
 2013-12-15 17:25:33 -08:00
Reynold Xin bad85b051d Use murmur3 hash for open hashset. 
(cherry picked from commit 212ff6834515543163aa63a3f4f762ebe641f8ca)
Signed-off-by: Ankur Dave <ankurdave@gmail.com>
 2013-12-15 17:23:15 -08:00
Reynold Xin 62bdc44a1e Unrolled while loop in readVarLong. 
(cherry picked from commit 45ffb1ae3c0527aae50502741a3585c411875b9a)
Signed-off-by: Ankur Dave <ankurdave@gmail.com>
 2013-12-15 17:23:15 -08:00
Ankur Dave 0459747c1c Fix typo 2013-12-14 17:08:04 -08:00
Ankur Dave 8a56c1ff67 Merge pull request #84 from amatsukawa/graphlab_enhancements 
GraphLab bug fix & set start vertex
 2013-12-14 16:29:24 -08:00
Ankur Dave 1ad3d240b0 Remove commented code 2013-12-14 15:49:00 -08:00
Ankur Dave 7dbd3bf825 Remove debug printing from PregelSuite 2013-12-14 15:45:23 -08:00
Ankur Dave e9cd634d05 Merge remote-tracking branch 'upstream/master' into mrTriplets-active-set 
Updates standalone PageRank to work with the new API.
 2013-12-14 15:44:35 -08:00
Ankur Dave 0c3fc1c1b6 Avoid re-creating the view RDDs multiple times 
Previously, successive operations that support incremental view
maintenance would inadvertently recreate previous view RDDs by calling
VTableReplicated.get(), which created the RDDs anew though they were
already cached. This change memoizes the RDDs and separates the process
of shipping active-set information to an existing view.
 2013-12-14 15:28:24 -08:00
Ankur Dave 59f625b745 Revert "Add debug logging to Pregel" 
This reverts commit e62013cdd92137507a96b6a5b347a1d239209587.
 2013-12-14 15:28:24 -08:00
Ankur Dave ee5c69e481 Fix bug in VertexPartition.isActive 
This took me ~5 hours to find!
 2013-12-14 15:28:23 -08:00
Ankur Dave 4d3bba3a13 Add debug logging to Pregel 2013-12-14 15:28:23 -08:00
Ankur Dave cf6288b993 Add PregelSuite 2013-12-14 15:28:23 -08:00
Ankur Dave 9c8b6224ec Remove unused bound variable 2013-12-14 15:28:23 -08:00
Ankur Dave ef17ab58cb Add another mrTriplets test 2013-12-14 15:28:23 -08:00
Ankur Dave b40824bc2b Allow innerJoining VertexPartitions with different indexes 2013-12-14 15:28:23 -08:00
Ankur Dave 7a8952e9bb Replace skipStale with activeSetOpt in mrTriplets (fails Pregel) 2013-12-14 15:28:23 -08:00
Ankur Dave 77b92748ad Replace update with innerJoin (has a bug) 
There is a conflict between vertices that didn't change so are not moved
but still need to run, and vertices that were deleted by the innerJoin
so should not run.
 2013-12-14 15:28:23 -08:00
Ankur Dave d161caa6eb Expose srcStale and dstStale 2013-12-14 15:28:23 -08:00
Ankur Dave 4bbae5c39b Remove printlns from test 2013-12-14 15:28:23 -08:00
Ankur Dave b2f595f552 Test more graph ops and skipStale 2013-12-14 15:28:22 -08:00
Ankur Dave 45ea674e21 Handle diffing VertexPartitions with different indexes 2013-12-14 15:28:22 -08:00
Ankur Dave 8f4b8e9b95 Reuse previous localVidMap if available 2013-12-14 15:28:22 -08:00
Ankur Dave 5e20cbaf66 Define localVidMap once per VTableReplicated 2013-12-14 15:28:22 -08:00
Ankur Dave dc72147900 Fix bug in interaction of incr. view maint., skipStale, and join rewrite 2013-12-14 15:28:22 -08:00
Ankur Dave d076fe5c9e Finish up 49f7fc5ed2edd42b118182438d84ed7d4e7d521f 2013-12-14 15:28:22 -08:00
Ankur Dave 4f4f0add24 Fix bug in mapVertices and outerJoinVertices 2013-12-14 15:28:22 -08:00
Ankur Dave 6bf51a5e15 Don't expose {src,dst}Mask in EdgeTriplet 
Exposing this information is not necessary due to the addition of
skipStale.
 2013-12-14 15:28:22 -08:00
Ankur Dave 87f2909561 Incremental view maintenance for all graph ops 
All GraphImpl operations now support incremental maintenance of the
replicated vertex view by reusing vTableReplicated whenever
possible. This has two consequences:

- This is not confined to the deltaJoinVertices operator anymore,
  so we rename it to the more intuitive "updateVertices" instead.

- This enables mrTriplets to support skipStale.
 2013-12-14 15:28:22 -08:00
Ankur Dave c3e0f01ad9 Remove calls to ClosureCleaner in GraphX 
All Spark RDD methods already clean closures so we don't need to.
 2013-12-14 15:28:21 -08:00
Ankur Dave 016cabceca Clean up imports in EdgeRDD 2013-12-14 15:03:57 -08:00
Ankur Dave d00cc8092b Fix argument bug and closure capture 2013-12-14 15:03:57 -08:00
Ankur Dave a8c7ebf0ed Don't partition edges by default; refactor 
Instead, expose Graph.partitionBy(PartitionStrategy).
 2013-12-14 15:03:57 -08:00
Ankur Dave 1e98840128 Load edges in columnar format 
In GraphLoader.edgeListFile, load edges directly into EdgePartitions,
avoiding repartitioning.
 2013-12-14 15:01:51 -08:00
Reynold Xin 9bf192b01c Merge pull request #91 from amplab/standalone-pagerank 
Standalone PageRank
 2013-12-14 12:52:18 -08:00
Reynold Xin 840af5e8e4 Merge pull request #99 from ankurdave/only-dynamic-pregel 
Remove static Pregel; take maxIterations in dynamic Pregel
 2013-12-14 12:51:51 -08:00
Ankur Dave 5b0d6f0ad5 Remove static Pregel; take maxIters in dynamic Pregel 2013-12-12 18:03:19 -08:00
Ankur Dave 3f69cdc81b Use standalone PageRank in Analytics 2013-12-12 15:43:12 -08:00
Ankur Dave a0fb477726 Test standalone PageRank 2013-12-12 15:42:55 -08:00
Ankur Dave e8ba51d644 Add standalone PageRank using only GraphX operators 2013-12-12 15:42:27 -08:00
Reynold Xin ce6ca4ea61 Merge pull request #97 from dcrankshaw/fix-rddtop 
Added BoundedPriorityQueue kryo registrator. Fixes top issue.
 2013-12-11 22:30:54 -08:00
Dan Crankshaw 12483d4ae6 Added BoundedPriorityQueue kryo registrator. Fixes top issue. 2013-12-11 20:56:21 -08:00
Reynold Xin 3abfbfb104 Merge pull request #92 from ankurdave/rdd-names 
Set RDD names for easy debugging
 2013-12-07 11:24:19 -08:00
Ankur Dave 84d0e1a334 Set RDD names for easy debugging 2013-12-07 04:05:45 -08:00
Ankur Dave 31e8a14e17 Merge pull request #90 from amplab/pregel-replicate-changed 
Replicate only changed vertices
 2013-12-06 21:49:55 -08:00
Reynold Xin 41721b1494 Fixed a bug in VTableReplicated that we only process the first block. 2013-12-06 00:51:12 -08:00
Reynold Xin 3b0ee53eda Minor update. 2013-12-05 23:30:56 -08:00
Reynold Xin 15168d6c4d Fixed a bug in VTableReplicated that we are always broadcasting all the vertices. 2013-12-05 23:25:53 -08:00
Reynold Xin a6075ba11f Merge branch 'pregel-replicate-changed' of github.com:ankurdave/graphx into pregel-replicate-changed 2013-12-05 22:35:16 -08:00
Ankur Dave b707861ba0 Simplify GraphImpl.deltaJoinVertices 2013-12-05 20:01:32 -08:00