ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5793 commits 6 branches 181 tags 397 MiB
Commit graph

5793 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ankur Dave 11dd35c28b Clean up GraphGenerators 2014-01-10 15:23:32 -08:00
Ankur Dave 9e48af6dba Remove unused HashUtils class 2014-01-10 15:22:57 -08:00
Ankur Dave b437ed62a8 graph -> graphx in pom.xml 2014-01-10 15:22:31 -08:00
Ankur Dave 7bda997785 Improve docs for PartitionStrategy 2014-01-10 13:00:28 -08:00
Ankur Dave eb4b46f8d1 Improve docs for GraphOps 2014-01-10 12:46:00 -08:00
Ankur Dave 9454fa1f6c Remove duplicate method in GraphLoader and improve docs 2014-01-10 12:37:20 -08:00
Ankur Dave 37611e57f6 Improve docs for EdgeRDD, EdgeTriplet, and GraphLab 2014-01-10 12:37:03 -08:00
Ankur Dave eee9bc0958 Remove commented-out perf files 2014-01-10 12:36:15 -08:00
Ankur Dave c39ec3017f Remove some commented code 2014-01-10 12:17:17 -08:00
Ankur Dave 5fcd2a61b4 Finish cleaning up Graph docs 2014-01-10 12:17:04 -08:00
Ankur Dave 4c114a7556 Start cleaning up Scaladocs in Graph and EdgeRDD 2014-01-10 11:37:54 -08:00
Ankur Dave 3eb83191cb Generate GraphX docs 2014-01-10 11:37:28 -08:00
Ankur Dave 6bd9a78e78 Add back Bagel links to docs, but mark them superseded 2014-01-10 11:37:10 -08:00
Ankur Dave cfc10c74a3 Remove EdgeTriplet.{src,dst}Stale, which were unused 2014-01-10 10:43:23 -08:00
Ankur Dave bf50e8c6cd Remove commented code from Analytics 2014-01-10 10:37:04 -08:00
Ankur Dave 1b2aad918c Update graphx/pom.xml to mirror mllib/pom.xml 2014-01-10 10:34:40 -08:00
Ankur Dave 23d2995116 Merge pull request #1 from jegonzal/graphx 
ProgrammingGuide
 2014-01-10 10:20:02 -08:00
Ankur Dave 729277ebc4 Undo 8b6b8ac87f 
Getting unpersist right in GraphLab is tricky.
 2014-01-10 01:53:28 -08:00
Ankur Dave 4cc550909a graph -> graphx in log4j.properties 2014-01-10 00:59:59 -08:00
Joseph E. Gonzalez b1eeefb401 WIP. Updating figures and cleaning up initial skeleton for GraphX Programming guide. 2014-01-10 00:39:08 -08:00
Ankur Dave ba511f890e Avoid recomputation by caching all multiply-used RDDs 2014-01-10 00:35:02 -08:00
Ankur Dave 8b6b8ac87f Unpersist previous iterations in GraphLab 2014-01-10 00:34:08 -08:00
Ankur Dave 2578332f97 Add Graph.unpersistVertices() 2014-01-09 23:34:35 -08:00
Ankur Dave 8ae108f6c4 Unpersist previous iterations in Pregel 2014-01-09 23:25:35 -08:00
Ankur Dave 210f2dd84f graph -> graphx in bin/compute-classpath.sh 2014-01-09 21:47:40 -08:00
Ankur Dave b7c92dded3 Add implicit algorithm methods for Graph; remove standalone PageRank 2014-01-09 20:44:28 -08:00
Ankur Dave 731f56f309 graph -> graphx 2014-01-09 14:31:33 -08:00
Ankur Dave 100718bcd3 Svdpp -> SVDPlusPlus 2014-01-09 14:05:09 -08:00
Ankur Dave 43e1bdc80c Pid -> PartitionID 2014-01-09 14:00:17 -08:00
Ankur Dave da83038234 Vid -> VertexID 2014-01-09 14:00:16 -08:00
Ankur Dave ec12c63409 Unwrap Graph.mapEdges signature 2014-01-09 13:29:20 -08:00
Ankur Dave 3b2e22e2c3 Revert changes to examples/.../PageRankUtils.scala 
Reverts to 04d83fc37f9eef89c20331c85291a0a169f75e6d:examples/src/main/scala/org/apache/spark/examples/bagel/PageRankUtils.scala.
 2014-01-09 13:27:40 -08:00
Ankur Dave 4aa9be1633 Make GraphImpl serializable to work around capture 2014-01-09 13:26:17 -08:00
Ankur Dave b5b0de2de5 Start fixing formatting of graphx-programming-guide 2014-01-09 13:24:25 -08:00
Ankur Dave e4483582fc Add docs/graphx-programming-guide.md from 7210257ba3038d5e22d4b60fe9c3113dc45c3dff:README.md 2014-01-09 10:24:43 -08:00
Ankur Dave 7309a29c75 Removed Kryo dependency and graphx-shell 2014-01-09 00:13:23 -08:00
Ankur Dave 22374559a2 Remove GraphX README 2014-01-08 22:48:54 -08:00
Ankur Dave 74fdfac112 Fix AbstractMethodError by inlining zip{Edge,Vertex}Partitions 
The zip{Edge,Vertex}Partitions methods created doubly-nested closures
and passed them to zipPartitions. For some reason this caused an
AbstractMethodError when zipPartitions tried to invoke the closure. This
commit works around the problem by inlining these methods wherever they
are called, eliminating the doubly-nested closure.
 2014-01-08 21:19:14 -08:00
Ankur Dave ab861d8450 Take SparkConf in constructor of Serializer subclasses 2014-01-08 21:19:14 -08:00
Ankur Dave 0ad75cdfb0 Manifest -> Tag in variable names 2014-01-08 21:19:14 -08:00
Ankur Dave ac536345f8 ClassManifest -> ClassTag 2014-01-08 21:19:14 -08:00
Ankur Dave 78d6b13ac8 Fix mis-merge in 44fd30d3fb 2014-01-08 21:19:14 -08:00
Ankur Dave 91227566bc Merge remote-tracking branch 'spark-upstream/master' into HEAD 
Conflicts:
	README.md
	core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
	core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
	core/src/main/scala/org/apache/spark/util/collection/PrimitiveKeyOpenHashMap.scala
	pom.xml
	project/SparkBuild.scala
	repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
 2014-01-08 21:19:08 -08:00
Reynold Xin 04d83fc37f Merge pull request #360 from witgo/master 
fix make-distribution.sh show version: command not found
 2014-01-08 11:55:37 -08:00
Reynold Xin 56ebfeaa52 Merge pull request #357 from hsaputra/set_boolean_paramname 
Set boolean param name for call to SparkHadoopMapReduceUtil.newTaskAttemptID

Set boolean param name for call to SparkHadoopMapReduceUtil.newTaskAttemptID to make it clear which param being set.
 2014-01-08 11:50:06 -08:00
Patrick Wendell bdeaeafbda Merge pull request #358 from pwendell/add-cdh 
Add CDH Repository to Maven Build

At some point this was removed from the Maven build... so I'm adding it back. It's needed for the Hadoop2 tests we run on Jenkins and it's also included in the SBT build.
 2014-01-08 11:48:39 -08:00
Reynold Xin 5cae05f59e Merge pull request #356 from hsaputra/remove_deprecated_cleanup_method 
Remove calls to deprecated mapred's OutputCommitter.cleanupJob

Since Hadoop 1.0.4 the mapred OutputCommitter.commitJob should do cleanup job via call to OutputCommitter.cleanupJob,

Remove SparkHadoopWriter.cleanup since it is used only by PairRDDFunctions.

In fact the implementation of mapred OutputCommitter.commitJob looks like this:

  public void commitJob(JobContext jobContext) throws IOException {
    cleanupJob(jobContext);
  }
 2014-01-08 11:47:28 -08:00
liguoqiang cf4aaf92d6 fix make-distribution.sh show version: command not found 2014-01-09 00:34:53 +08:00
Thomas Graves 6eef78d769 Merge pull request #345 from colorant/yarn 
support distributing extra files to worker for yarn client mode

So that user doesn't need to package all dependency into one assemble jar as spark app jar
 2014-01-08 08:49:20 -06:00
Patrick Wendell 3209a86f39 Add CDH Repository to Maven Build 2014-01-08 01:21:17 -08:00