ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5203 commits 6 branches 181 tags 397 MiB
Commit graph

376 commits

Author SHA1 Message Date
Patrick Wendell c78b381e91 Fixes 2014-01-08 00:09:12 -08:00
Patrick Wendell b72cceba27 Some doc fixes 2014-01-06 22:05:53 -08:00
Patrick Wendell c0498f9265 Merge remote-tracking branch 'apache-github/master' into standalone-driver 
Conflicts:
	core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala
	core/src/main/scala/org/apache/spark/deploy/client/TestClient.scala
	core/src/main/scala/org/apache/spark/deploy/master/Master.scala
	core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
	core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
 2014-01-06 17:29:21 -08:00
Andrew Ash 2dd4fb5698 Clarify spark.cores.max 
It controls the count of cores across the cluster, not on a per-machine basis.
 2014-01-06 09:01:46 -08:00
Patrick Wendell 604fad9c39 Merge remote-tracking branch 'apache-github/master' into remove-binaries 
Conflicts:
	core/src/test/scala/org/apache/spark/DriverSuite.scala
	docs/python-programming-guide.md
 2014-01-03 21:29:33 -08:00
Patrick Wendell 4ae101ff38 Merge pull request #317 from ScrapCodes/spark-915-segregate-scripts 
Spark-915 segregate scripts
 2014-01-03 11:24:35 -08:00
Prashant Sharma 74ba97fcf7 sbin/spark-class* -> bin/spark-class* 2014-01-03 15:08:01 +05:30
Prashant Sharma 94f2fffa23 fixed review comments 2014-01-03 14:43:37 +05:30
Prashant Sharma b4bb80002b Merge branch 'master' into spark-1002-remove-jars 2014-01-03 12:12:04 +05:30
Raymond Liu f442afc22e fix docs for yarn 2014-01-03 14:14:35 +08:00
Raymond Liu ebdfa6bb97 Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2 2014-01-03 12:14:38 +08:00
Raymond Liu 7815a3ace9 Update maven build documentation 2014-01-03 12:12:38 +08:00
Raymond Liu be343d2a56 Fix yarn/README.md and update docs/running-on-yarn.md 2014-01-03 12:12:38 +08:00
Prashant Sharma 59e8009b8d a few left over document change 2014-01-02 21:48:44 +05:30
Prashant Sharma a3f90a2ecf pyspark -> bin/pyspark 2014-01-02 18:50:12 +05:30
Prashant Sharma 94b7a7fe37 run-example -> bin/run-example 2014-01-02 18:41:21 +05:30
Prashant Sharma b810a85cdd spark-shell -> bin/spark-shell 2014-01-02 18:37:40 +05:30
Prashant Sharma 980afd280a Merge branch 'scripts-reorg' of github.com:shane-huang/incubator-spark into spark-915-segregate-scripts 
Conflicts:
	bin/spark-shell
	core/pom.xml
	core/src/main/scala/org/apache/spark/SparkContext.scala
	core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
	core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala
	core/src/test/scala/org/apache/spark/DriverSuite.scala
	python/run-tests
	sbin/compute-classpath.sh
	sbin/spark-class
	sbin/stop-slaves.sh
 2014-01-02 17:55:21 +05:30
Prashant Sharma 6be4c11194 Removed sbt folder and changed docs accordingly 2014-01-02 14:09:37 +05:30
Matei Zaharia 0fa5809768 Updated docs for SparkConf and handled review comments 2013-12-30 22:17:28 -05:00
Patrick Wendell 6ffa9bb226 Documentation and adding supervise option 2013-12-29 11:26:56 -08:00
Reynold Xin 72a17b69f5 Revert "Merge pull request #310 from jyunfan/master" 
This reverts commit 79b20e4dbe, reversing
changes made to 7375047d51.
 2013-12-28 21:25:40 -10:00
Jyun-Fan Tsai 17f6620a71 Fix typo in the Accumulators section 
val => var
 2013-12-29 11:30:02 +08:00
fengdong ad8ce0148a changed the example links in the scala-programming-guid 2013-12-18 19:03:32 +08:00
fengdong ddebaf8280 Fixed the example link. 2013-12-18 11:00:36 +08:00
Reynold Xin 7db9165961 Merge pull request #251 from pwendell/master 
Fix list rendering in YARN markdown docs.

This is some minor clean-up which makes the list render correctly.
 2013-12-14 14:16:34 -08:00
Prashant Sharma d3090b79a5 A few corrections to documentation. 2013-12-12 10:12:06 +05:30
Prashant Sharma 603af51bb5 Merge branch 'master' into akka-bug-fix 
Conflicts:
	core/pom.xml
	core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
	pom.xml
	project/SparkBuild.scala
	streaming/pom.xml
	yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
 2013-12-11 10:21:53 +05:30
Patrick Wendell 1291dd4dce Fix list rendering in YARN markdown docs. 2013-12-10 16:38:33 -08:00
Patrick Wendell 0428145ed4 Small fix 2013-12-07 22:33:11 -08:00
Patrick Wendell b3e87c0f51 Adding HDP 2.0 version 2013-12-07 22:31:46 -08:00
Patrick Wendell 41c60b337a Various broken links in documentation 2013-12-07 22:31:44 -08:00
Patrick Wendell 6494d62fe4 Merge pull request #240 from pwendell/master 
SPARK-917 Improve API links in nav bar
 2013-12-07 11:56:16 -08:00
Patrick Wendell dd331a6b26 SPARK-917 Improve API links in nav bar 2013-12-07 11:49:49 -08:00
Aaron Davidson cb6ac8aafb Correct spellling error in configuration.md 2013-12-07 01:40:01 -08:00
Patrick Wendell 7a1d1c93b8 Minor formatting fix in config file 2013-12-06 20:28:22 -08:00
Patrick Wendell 1b38f5f277 Merge pull request #236 from pwendell/shuffle-docs 
Adding disclaimer for shuffle file consolidation
 2013-12-06 20:16:15 -08:00
Patrick Wendell b9451acdf4 Adding disclaimer for shuffle file consolidation 2013-12-06 19:25:28 -08:00
Patrick Wendell bb6e25c663 Minor doc fixes and updating README 2013-12-06 17:42:28 -08:00
Ali Ghodsi e2c2914faa more docs 2013-12-06 16:54:06 -08:00
Ali Ghodsi f2fb4b4228 Updated documentation about the YARN v2.2 build process 2013-12-06 16:31:26 -08:00
Patrick Wendell 5d460253d6 Merge pull request #228 from pwendell/master 
Document missing configs and set shuffle consolidation to false.
 2013-12-05 12:31:24 -08:00
Patrick Wendell 1450b8ef87 Small changes from Matei review 2013-12-04 18:49:32 -08:00
Patrick Wendell b1c6fa1584 Document missing configs and set shuffle consolidation to false. 2013-12-04 18:39:34 -08:00
Andrew Ash 0c5af38b86 Typo: applicaton 2013-12-04 12:30:25 -08:00
Prashant Sharma 17987778da Merge branch 'master' into wip-scala-2.10 
Conflicts:
	core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
	core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala
	core/src/main/scala/org/apache/spark/rdd/MapPartitionsWithContextRDD.scala
	core/src/main/scala/org/apache/spark/rdd/RDD.scala
	python/pyspark/rdd.py
 2013-11-27 14:44:12 +05:30
Prashant Sharma 54862af5ee Improvements from the review comments and followed Boy Scout Rule. 2013-11-27 14:26:28 +05:30
Prashant Sharma dca946ff67 Documenting the newly added spark properties. 2013-11-26 20:47:38 +05:30
Andrew Ash 08afef37a0 Update tuning.md 
Clarify when serializer is used based on recent user@ mailing list discussion.
 2013-11-25 17:08:52 -08:00
Matei Zaharia eb4296c8f7 Merge pull request #101 from colorant/yarn-client-scheduler 
For SPARK-527, Support spark-shell when running on YARN

sync to trunk and resubmit here

In current YARN mode approaching, the application is run in the Application Master as a user program thus the whole spark context is on remote.

This approaching won't support application that involve local interaction and need to be run on where it is launched.

So In this pull request I have a YarnClientClusterScheduler and backend added.

With this scheduler, the user application is launched locally,While the executor will be launched by YARN on remote nodes with a thin AM which only launch the executor and monitor the Driver Actor status, so that when client app is done, it can finish the YARN Application as well.

This enables spark-shell to run upon YARN.

This also enable other Spark applications to have the spark context to run locally with a master-url "yarn-client". Thus e.g. SparkPi could have the result output locally on console instead of output in the log of the remote machine where AM is running on.

Docs also updated to show how to use this yarn-client mode.
 2013-11-25 15:25:29 -08:00