ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5025 commits 6 branches 181 tags 397 MiB
Commit graph

319 commits

Author SHA1 Message Date
Ankur Dave 9193a8f788 Merge remote-tracking branch 'upstream/master' into add_project_to_graph 
Conflicts:
	graph/src/main/scala/org/apache/spark/graph/Graph.scala
	graph/src/main/scala/org/apache/spark/graph/impl/GraphImpl.scala
 2013-12-18 13:00:58 -08:00
Joey 3fd2e09ffb Merge pull request #104 from jianpingjwang/master 
SVD++ demo
 2013-12-18 12:52:36 -08:00
Wang Jianping J 06581b6a96 Update Svdpp.scala 2013-12-18 06:51:37 +08:00
Wang Jianping J ba538797dd Update Svdpp.scala 2013-12-18 06:42:19 +08:00
Wang Jianping J e94fe39d0f Update Svdpp.scala 2013-12-18 06:39:28 +08:00
Wang Jianping J 772b192910 Update AnalyticsSuite.scala 2013-12-17 19:41:04 +08:00
Wang Jianping J 9d2351f501 Update Svdpp.scala 2013-12-17 19:30:07 +08:00
Wang Jianping J aee71156f3 Test SVD++ 2013-12-17 19:20:25 +08:00
Wang Jianping J fb1d1e45f1 Update Svdpp.scala 2013-12-17 19:17:40 +08:00
Ankur Dave 3ade8be8f2 Add clustered index on edges by source vertex 
This allows efficient edge scan in mapReduceTriplets when many source
vertices are inactive. The scan method switches from edge scan to
clustered index scan when less than 80% of source vertices are active.
 2013-12-16 17:37:51 -08:00
Ankur Dave 9df565007b Merge remote-tracking branch 'upstream/master' into mrTriplets-active-set 
Conflicts:
	graph/src/main/scala/org/apache/spark/graph/impl/GraphImpl.scala
 2013-12-16 16:24:43 -08:00
Joey 5192ef3859 Merge pull request #94 from ankurdave/load-edges-columnar 
Load edges in columnar format
 2013-12-16 15:08:08 -08:00
Wang Jianping J 5eb21e6720 SVD++ demo 2013-12-16 20:01:31 +08:00
Reynold Xin ded10ce5b0 Merge pull request #103 from amplab/optimizations 
Optimizations cherry-picked from SIGMOD branches
 2013-12-15 17:25:33 -08:00
Reynold Xin 62bdc44a1e Unrolled while loop in readVarLong. 
(cherry picked from commit 45ffb1ae3c0527aae50502741a3585c411875b9a)
Signed-off-by: Ankur Dave <ankurdave@gmail.com>
 2013-12-15 17:23:15 -08:00
Ankur Dave 0459747c1c Fix typo 2013-12-14 17:08:04 -08:00
Ankur Dave 8a56c1ff67 Merge pull request #84 from amatsukawa/graphlab_enhancements 
GraphLab bug fix & set start vertex
 2013-12-14 16:29:24 -08:00
Ankur Dave 1ad3d240b0 Remove commented code 2013-12-14 15:49:00 -08:00
Ankur Dave 7dbd3bf825 Remove debug printing from PregelSuite 2013-12-14 15:45:23 -08:00
Ankur Dave e9cd634d05 Merge remote-tracking branch 'upstream/master' into mrTriplets-active-set 
Updates standalone PageRank to work with the new API.
 2013-12-14 15:44:35 -08:00
Ankur Dave 0c3fc1c1b6 Avoid re-creating the view RDDs multiple times 
Previously, successive operations that support incremental view
maintenance would inadvertently recreate previous view RDDs by calling
VTableReplicated.get(), which created the RDDs anew though they were
already cached. This change memoizes the RDDs and separates the process
of shipping active-set information to an existing view.
 2013-12-14 15:28:24 -08:00
Ankur Dave 59f625b745 Revert "Add debug logging to Pregel" 
This reverts commit e62013cdd92137507a96b6a5b347a1d239209587.
 2013-12-14 15:28:24 -08:00
Ankur Dave ee5c69e481 Fix bug in VertexPartition.isActive 
This took me ~5 hours to find!
 2013-12-14 15:28:23 -08:00
Ankur Dave 4d3bba3a13 Add debug logging to Pregel 2013-12-14 15:28:23 -08:00
Ankur Dave cf6288b993 Add PregelSuite 2013-12-14 15:28:23 -08:00
Ankur Dave 9c8b6224ec Remove unused bound variable 2013-12-14 15:28:23 -08:00
Ankur Dave ef17ab58cb Add another mrTriplets test 2013-12-14 15:28:23 -08:00
Ankur Dave b40824bc2b Allow innerJoining VertexPartitions with different indexes 2013-12-14 15:28:23 -08:00
Ankur Dave 7a8952e9bb Replace skipStale with activeSetOpt in mrTriplets (fails Pregel) 2013-12-14 15:28:23 -08:00
Ankur Dave 77b92748ad Replace update with innerJoin (has a bug) 
There is a conflict between vertices that didn't change so are not moved
but still need to run, and vertices that were deleted by the innerJoin
so should not run.
 2013-12-14 15:28:23 -08:00
Ankur Dave d161caa6eb Expose srcStale and dstStale 2013-12-14 15:28:23 -08:00
Ankur Dave 4bbae5c39b Remove printlns from test 2013-12-14 15:28:23 -08:00
Ankur Dave b2f595f552 Test more graph ops and skipStale 2013-12-14 15:28:22 -08:00
Ankur Dave 45ea674e21 Handle diffing VertexPartitions with different indexes 2013-12-14 15:28:22 -08:00
Ankur Dave 8f4b8e9b95 Reuse previous localVidMap if available 2013-12-14 15:28:22 -08:00
Ankur Dave 5e20cbaf66 Define localVidMap once per VTableReplicated 2013-12-14 15:28:22 -08:00
Ankur Dave dc72147900 Fix bug in interaction of incr. view maint., skipStale, and join rewrite 2013-12-14 15:28:22 -08:00
Ankur Dave d076fe5c9e Finish up 49f7fc5ed2edd42b118182438d84ed7d4e7d521f 2013-12-14 15:28:22 -08:00
Ankur Dave 4f4f0add24 Fix bug in mapVertices and outerJoinVertices 2013-12-14 15:28:22 -08:00
Ankur Dave 6bf51a5e15 Don't expose {src,dst}Mask in EdgeTriplet 
Exposing this information is not necessary due to the addition of
skipStale.
 2013-12-14 15:28:22 -08:00
Ankur Dave 87f2909561 Incremental view maintenance for all graph ops 
All GraphImpl operations now support incremental maintenance of the
replicated vertex view by reusing vTableReplicated whenever
possible. This has two consequences:

- This is not confined to the deltaJoinVertices operator anymore,
  so we rename it to the more intuitive "updateVertices" instead.

- This enables mrTriplets to support skipStale.
 2013-12-14 15:28:22 -08:00
Ankur Dave c3e0f01ad9 Remove calls to ClosureCleaner in GraphX 
All Spark RDD methods already clean closures so we don't need to.
 2013-12-14 15:28:21 -08:00
Ankur Dave 016cabceca Clean up imports in EdgeRDD 2013-12-14 15:03:57 -08:00
Ankur Dave d00cc8092b Fix argument bug and closure capture 2013-12-14 15:03:57 -08:00
Ankur Dave a8c7ebf0ed Don't partition edges by default; refactor 
Instead, expose Graph.partitionBy(PartitionStrategy).
 2013-12-14 15:03:57 -08:00
Ankur Dave 1e98840128 Load edges in columnar format 
In GraphLoader.edgeListFile, load edges directly into EdgePartitions,
avoiding repartitioning.
 2013-12-14 15:01:51 -08:00
Reynold Xin 9bf192b01c Merge pull request #91 from amplab/standalone-pagerank 
Standalone PageRank
 2013-12-14 12:52:18 -08:00
Ankur Dave 5b0d6f0ad5 Remove static Pregel; take maxIters in dynamic Pregel 2013-12-12 18:03:19 -08:00
Ankur Dave 3f69cdc81b Use standalone PageRank in Analytics 2013-12-12 15:43:12 -08:00
Ankur Dave a0fb477726 Test standalone PageRank 2013-12-12 15:42:55 -08:00