ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5273 commits 6 branches 181 tags 397 MiB
Commit graph

389 commits

Author SHA1 Message Date
Patrick Wendell 112c0a1776 Fixing config option "retained_stages" => "retainedStages". 
This is a very esoteric option and it's out of sync with the style we use.
So it seems fitting to fix it for 0.9.0.
 2014-01-08 21:16:16 -08:00
Thomas Graves 6eef78d769 Merge pull request #345 from colorant/yarn 
support distributing extra files to worker for yarn client mode

So that user doesn't need to package all dependency into one assemble jar as spark app jar
 2014-01-08 08:49:20 -06:00
Patrick Wendell bb6a39a687 Merge pull request #322 from falaki/MLLibDocumentationImprovement 
SPARK-1009 Updated MLlib docs to show how to use it in Python

In addition added detailed examples for regression, clustering and recommendation algorithms in a separate Scala section. Fixed a few minor issues with existing documentation.
 2014-01-07 22:32:18 -08:00
Hossein Falaki 46cb980a5f Fixed merge conflict 2014-01-07 21:28:26 -08:00
Matei Zaharia 2c421749ea Address review comments 2014-01-07 19:30:23 -05:00
Matei Zaharia d8bcc8e9a0 Add way to limit default # of cores used by applications on standalone mode 
Also documents the spark.deploy.spreadOut option.
 2014-01-07 14:35:52 -05:00
Patrick Wendell c3cf0475e8 Merge pull request #339 from ScrapCodes/conf-improvements 
Conf improvements

There are two new features.

1. Allow users to set arbitrary akka configurations via spark conf.

2. Allow configuration to be printed in logs for diagnosis.
 2014-01-07 00:54:25 -08:00
Reynold Xin a862cafacf Merge pull request #331 from holdenk/master 
Add a script to download sbt if not present on the system

As per the discussion on the dev mailing list this script will use the system sbt if present or otherwise attempt to install the sbt launcher. The fall back error message in the event it fails instructs the user to install sbt. While the URLs it fetches from aren't controlled by the spark project directly, they are stable and the current authoritative sources.
 2014-01-07 00:18:20 -08:00
Prashant Sharma c729fa7c8e formatting related fixes suggested by Patrick. 2014-01-07 13:08:16 +05:30
Prashant Sharma b84dc780d3 Allow configuration to be printed in logs for diagnosis. 2014-01-07 13:01:43 +05:30
Prashant Sharma b3018811e1 Allow users to set arbitrary akka configurations via spark conf. 2014-01-07 13:01:43 +05:30
Raymond Liu 67af803136 Export --file for YarnClient mode to support sending extra files to worker on yarn cluster 2014-01-07 10:24:11 +08:00
Hossein Falaki 150089dae1 Added proper evaluation example for collaborative filtering and fixed typo 2014-01-06 12:43:17 -08:00
Andrew Ash 2dd4fb5698 Clarify spark.cores.max 
It controls the count of cores across the cluster, not on a per-machine basis.
 2014-01-06 09:01:46 -08:00
Holden Karau d86dc74d79 Code review feedback 2014-01-05 22:05:30 -08:00
Patrick Wendell 604fad9c39 Merge remote-tracking branch 'apache-github/master' into remove-binaries 
Conflicts:
	core/src/test/scala/org/apache/spark/DriverSuite.scala
	docs/python-programming-guide.md
 2014-01-03 21:29:33 -08:00
Hossein Falaki 8b5be06752 Added table of contents and minor fixes 2014-01-03 16:38:33 -08:00
Patrick Wendell 4ae101ff38 Merge pull request #317 from ScrapCodes/spark-915-segregate-scripts 
Spark-915 segregate scripts
 2014-01-03 11:24:35 -08:00
Prashant Sharma 74ba97fcf7 sbin/spark-class* -> bin/spark-class* 2014-01-03 15:08:01 +05:30
Prashant Sharma 94f2fffa23 fixed review comments 2014-01-03 14:43:37 +05:30
Prashant Sharma b4bb80002b Merge branch 'master' into spark-1002-remove-jars 2014-01-03 12:12:04 +05:30
Raymond Liu f442afc22e fix docs for yarn 2014-01-03 14:14:35 +08:00
Raymond Liu ebdfa6bb97 Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2 2014-01-03 12:14:38 +08:00
Raymond Liu 7815a3ace9 Update maven build documentation 2014-01-03 12:12:38 +08:00
Raymond Liu be343d2a56 Fix yarn/README.md and update docs/running-on-yarn.md 2014-01-03 12:12:38 +08:00
Hossein Falaki 81989e2664 Commented the last part of collaborative filtering examples that lead to errors 2014-01-02 16:22:13 -08:00
Hossein Falaki c189c8362c Added Scala and Python examples for mllib 2014-01-02 15:22:20 -08:00
Prashant Sharma 59e8009b8d a few left over document change 2014-01-02 21:48:44 +05:30
Prashant Sharma a3f90a2ecf pyspark -> bin/pyspark 2014-01-02 18:50:12 +05:30
Prashant Sharma 94b7a7fe37 run-example -> bin/run-example 2014-01-02 18:41:21 +05:30
Prashant Sharma b810a85cdd spark-shell -> bin/spark-shell 2014-01-02 18:37:40 +05:30
Prashant Sharma 980afd280a Merge branch 'scripts-reorg' of github.com:shane-huang/incubator-spark into spark-915-segregate-scripts 
Conflicts:
	bin/spark-shell
	core/pom.xml
	core/src/main/scala/org/apache/spark/SparkContext.scala
	core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
	core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala
	core/src/test/scala/org/apache/spark/DriverSuite.scala
	python/run-tests
	sbin/compute-classpath.sh
	sbin/spark-class
	sbin/stop-slaves.sh
 2014-01-02 17:55:21 +05:30
Prashant Sharma 6be4c11194 Removed sbt folder and changed docs accordingly 2014-01-02 14:09:37 +05:30
Matei Zaharia 0fa5809768 Updated docs for SparkConf and handled review comments 2013-12-30 22:17:28 -05:00
Reynold Xin 72a17b69f5 Revert "Merge pull request #310 from jyunfan/master" 
This reverts commit 79b20e4dbe, reversing
changes made to 7375047d51.
 2013-12-28 21:25:40 -10:00
Jyun-Fan Tsai 17f6620a71 Fix typo in the Accumulators section 
val => var
 2013-12-29 11:30:02 +08:00
fengdong ad8ce0148a changed the example links in the scala-programming-guid 2013-12-18 19:03:32 +08:00
fengdong ddebaf8280 Fixed the example link. 2013-12-18 11:00:36 +08:00
Reynold Xin 7db9165961 Merge pull request #251 from pwendell/master 
Fix list rendering in YARN markdown docs.

This is some minor clean-up which makes the list render correctly.
 2013-12-14 14:16:34 -08:00
Prashant Sharma d3090b79a5 A few corrections to documentation. 2013-12-12 10:12:06 +05:30
Prashant Sharma 603af51bb5 Merge branch 'master' into akka-bug-fix 
Conflicts:
	core/pom.xml
	core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
	pom.xml
	project/SparkBuild.scala
	streaming/pom.xml
	yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
 2013-12-11 10:21:53 +05:30
Patrick Wendell 1291dd4dce Fix list rendering in YARN markdown docs. 2013-12-10 16:38:33 -08:00
Patrick Wendell 0428145ed4 Small fix 2013-12-07 22:33:11 -08:00
Patrick Wendell b3e87c0f51 Adding HDP 2.0 version 2013-12-07 22:31:46 -08:00
Patrick Wendell 41c60b337a Various broken links in documentation 2013-12-07 22:31:44 -08:00
Patrick Wendell 6494d62fe4 Merge pull request #240 from pwendell/master 
SPARK-917 Improve API links in nav bar
 2013-12-07 11:56:16 -08:00
Patrick Wendell dd331a6b26 SPARK-917 Improve API links in nav bar 2013-12-07 11:49:49 -08:00
Aaron Davidson cb6ac8aafb Correct spellling error in configuration.md 2013-12-07 01:40:01 -08:00
Patrick Wendell 7a1d1c93b8 Minor formatting fix in config file 2013-12-06 20:28:22 -08:00
Patrick Wendell 1b38f5f277 Merge pull request #236 from pwendell/shuffle-docs 
Adding disclaimer for shuffle file consolidation
 2013-12-06 20:16:15 -08:00